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Making the Most of Parent Involvement Data: 
Improving Quality and Enhancing Understanding 


Introduction 


Parent involvement? is one of the most important predictors of students’ 
educational success. Compared to students whose parents are not involved in 
their education, students whose parents participate in their education tend to 


e Have better grades 

e Stay in school 

e Graduate from high school at higher rates 

e Demonstrate better social skills and behavior? 


It is important that schools take actions to facilitate this involvement.? 
Importantly, “the education of children with disabilities can be made more 
effective by...strengthening the role and responsibility of parents and ensuring 
that families...have meaningful opportunities to participate in the education of 
their children at school and at home.” 


The Individuals with Disabilities Education Act (IDEA) states that schools must 
give parents the opportunity to be involved in 


e Determining their children’s eligibility for services based on assessment 
and evaluation results 


e Determining appropriate educational placement 


e Serving as individualized education program (IEP) team members for 
their children? 


Each year, to comply with requirements of IDEA, states must submit to the 
U.S. Department of Education’s Office of Special Education Programs (OSEP) 
their State Performance Plan/Annual Performance Report (SPP/APR), which 
outlines their progress on a series of indicators related to improving outcomes 
for children with disabilities and their families. One part of this is related to 


Indicator B8, described in Box 1, which relates to parent involvement. In addition 


A Note On This Toolkit 


This toolkit is designed to assist 
states as they plan for and carry out 
their efforts to collect, report, and 
use high-quality parent involvement 
data. It defines key concepts; offers 
guidance on ways to improve the 
quality of the collection, analysis, 
and use of parent involvement data; 
and provides resources and tools to 
help states in their efforts. 


The toolkit is designed to allow 
various navigation options. There 
are four primary sections: preparing 
for data collection (READY), 
collecting data (RUN), analyzing data 
(REFLECT), and using the collected 
information to consider current 
results and goals and share data 
with others (REACH). In addition, 
there are links to other resources 
and “deeper dive” sections that 
provide additional information 
states may find helpful. States can 
proceed through the sections ina 
linear manner or use the navigation 
pane to go to individual sections. 


to helping states report on Indicator B8, collecting data on parent involvement is an important step in the process of 
determining how to improve the programs and services offered to children with disabilities and their families. 


1 The term parent involvement includes family involvement. Parent involvement is used throughout this guide, as it is the term used in State Performance Plan 


(SPP)/Annual Performance Report (APR) Indicator B8. 
2 Castro et al. 2015; Henderson and Mapp 2002; Van Voorhis et al. 2013; Wilder 2013. 
3 Henderson and Mapp 2002. 
4 Individuals with Disabilities Education Improvement Act of 2004, P.L. 108-446, Section 601(c)(5). 
5 Ibid. 
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What is Indicator B8? 


Indicator B8 measures the extent to which parents perceive that schools facilitate parent involvement as a means of 
improving services and results for children with disabilities. Box 1 outlines OSEP’s instructions regarding the data source 
and measurement states should use to report on Indicator B8.° 


Box 1. Requirements for Indicator B8 


FFY 2013-2018 Part B SPP/APR Indicator Measurement Table—indicator 8 

Indicator 8: Percent of parents with a child receiving special education services who report that schools facilitated 
parent involvement as a means of improving services and results for children with disabilities (20 U.S.C. 
§1416(a)(3)(A)). 


Data Source: State-selected data source. Measurement: Percent = number of respondent parents who report that 
schools facilitated parent involvement as a means of improving services and results for children with disabilities 
divided by the total number of respondent parents of children with disabilities, times 100. 


Instructions for Indicator/Measurement: Sampling of parents from whom response is requested is allowed. When 
sampling is used, submit a description of the sampling methodology outlining how the design will yield valid and 
reliable estimates. Describe the results of the calculations and compare the results to the target. Include a 
description of how the state has ensured that the response data are valid and reliable, including how the data 
represent the demographics of the state. Provide the actual numbers used in the calculation. 


Why is it important to improve the quality of Indicator B8 data? 


Given the importance of parent involvement to students’ educational outcomes, it is essential that states collect and 
report high-quality data related to that involvement and that the key audiences use those data effectively. Doing so will 
inform state and local education agencies on how they can improve the programs and services offered to students with 
disabilities and their families. Indicator B8 requires all states to collect and report parent involvement data—an 
important part of ensuring that parents of children with disabilities are given the opportunity to participate in their 
children’s education. Yet, a 2014 report published by the U.S. Government Accountability Office (GAO) found that 
because of differences in the Indicator B8 data collection and analysis methods used by states, OSEP cannot determine 
states’ individual performance or compare states’ performance in facilitating parent involvement. This variation across 
states limits OSEP’s ability to oversee state dispute resolution activities and monitor the level of parent involvement for 
issues that may require improvement activities. 7 Perhaps more importantly, issues with the quality of parent 
involvement data that are collected and reported to key stakeholders at state and local levels means that opportunities 
to increase parent involvement, and thereby improve educational outcomes for students with disabilities, may be 
missed. 


In response to the GAO report, OSEP agreed to take steps to improve the comparability of parent involvement data, 
while minimizing the burden on states. OSEP directed the /DEA Data Center (IDC) and other national technical assistance 
(TA) centers to develop and widely distribute (a) exemplars of best practices for collecting parent involvement data, 
including model surveys and methodologies, and (b) materials to assist states in analyzing and using parent involvement 
data to improve the provision of special education services. 


® U.S. Department of Education 2014. 
7U.S. Government Accountability Office 2014. 
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What are states currently doing to collect and report on Indicator B8? 


Currently, all states are conducting surveys to gather their Indicator B8 data; these are outlined in Table 1. 


Table 1. Types of surveys used to collect parent involvement data 


Survey Description How states have used the survey 
NCSEAM The NCSEAM scale measures schools’ Approximately 40 percent of states have reported using 
The National Center for efforts to facilitate parent involvement the NCSEAM.? 
Special Education and is recognized as a high-quality tool 


Most states using the NCSEAM have used the 25-item 


A ntability Monitorin for collectin rel Indi s 
ccountability Monitoring or collecting data related to Indicator Variohor the NCSCAM scale» 


(NCSEAM) developed the B8. NCSEAM recommended a 25-item 
Schools’ Efforts to Partner version of the scale in 2006, although 
with Parents Scale (SEPPS) there are 78 items that states can select 


specifically to meet the from an “item bank.” The complete list 
requirements of Indicator of the 78 validated items, along with a 

B8. The scale is commonly full description of the development of 

referred to as the the scale and its psychometric 


“NCSEAM.” characteristics, is included in Elbaum, 
Fisher, and Coulter (2011). Elbaum 
(2014) includes the 25-item version of 


the scale. 

State-developed surveys Many states have created their own Approximately 40 percent of states have reported using 
surveys to gather parent involvement a state-developed survey, and an additional 11 percent 
data, many of which are adapted from have reported using a modified version of the NCSEAM 
the NCSEAM scale. scale or the Family Outcomes Survey, or ECO.° 4 

States have used different approaches to developing 
surveys: 


e Adding new items to the NCSEAM 

e Adding selected NCSEAM items to a new survey 

e Adapting the ECO survey so it can be used with both 
a 3- to 5-year-old population and a school-aged 
population 

e Creating a new survey 


? Elbaum 2014. 

>. Elbaum, Fisher, and Coulter 2011. 

“ The Family Outcomes Survey was developed by the Early Childhood Outcomes (ECO) Center. This survey is commonly referred to as the “ECO.” 
4. U.S. Department of Education 2016. 
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READY: How can my state improve the quality of its parent 


involvement data? 


Collecting, analyzing, and using parent involvement data can be complex and challenging, and the process requires 

resources and support from personnel at the school, district, and state levels. Some states have internal personnel who 
work to collect, analyze, and report their parent involvement data, others collaborate with key stakeholder groups, and 
still others hire third-party contractors to conduct some or all of their parent involvement data collection and reporting 


activities. 


Consider the elements of data quality when planning data collection 


As your state begins planning its parent involvement data collection and analysis activities, you should consider what it 
means to collect high-quality data. Table 2 outlines five characteristics of high-quality data, lists questions you should 
ask about the data, and offers some examples related to collecting parent involvement data.°® 


Table 2. Elements of high-quality data to keep in mind as you plan to collect data 


Element 
Precision ° 


Questions to ask 
To what extent do the data collected 
reflect an exact measurement (for 


to respond to a question? 
Accuracy © 


e Do the data provide a “true” (e.g., 


experience? 
Reliability 0 


falsely? 


Consistency e 
responses agree with each other? 


Completeness | ° 
provided? 


quantitative data) or include the narrative 
information (for qualitative data) needed 


To what extent do data reflect the actual 
value of an observation or achievement? 


verifiable) account of a phenomena or 

To what extent can a data measurement be 
replicated with accuracy and precision? 

e When gathering data from individuals, 


might a person have any reason to respond 


To what extent do data or individuals’ 


To what extent is complete information 


Examples of parent involvement focused questions 
Does your survey ask parents to give a specific number 
(e.g., 5) or range (e.g., 3-5) of times they have actively 
participated in meetings related to their child’s education? 


In the survey responses, is the number of times parents 
report being invited to participate in individualized 
education program (IEP) meetings “off” by one or more 
meetings (when compared to additional data collected 
from school staff)? 

Does the survey produce similar results when administered 
to groups of parents who would be expected to respond in 
similar ways? 

Might parent focus group participants exaggerate the level 
of parent involvement at a particular school? 

When you compare responses to your parent survey with 
data about parent involvement provided by the school 
district or data collected through a teacher focus group, is 
the level of reported parent involvement similar across the 
different data sources? 

When a survey asks how often parents meet with their 
child’s teachers, do the responses provide detailed 
information such as 2-3 times per month? 

Is there enough context provided during an interview to 
understand a mother’s statement that her child’s school 
“has no interest in getting parents involved”? 


8 Adapted from Lammert, Heinemeier, Howell et al. 2016. 
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Choose data collection methods that will produce high-quality and rich data 


As you’ve learned, all states currently use surveys as their primary collection method for Indicator B8 data. But keep in 
mind that states are not required to use a particular method!! Your state may be interested in exploring other options 
due to cost considerations or a desire for more in-depth information. Additional data collection methods may include 


e Conducting interviews or focus groups with parents 


e Conducting secondary data analysis using data maintained in state and district data systems related to parent 
involvement (e.g., data on disputes or complaints, parent attendance at events, and active participation in the 
individualized education program [IEP] process) 


e Observations of parent involvement activities using rubrics completed by schools or districts with indicators of 
parent involvement, including opportunities for parent involvement, type and level of parent involvement, and 
partnerships between parents and teachers to jointly set goals for students 


This toolkit focuses primarily on the use of parent surveys to collect parent involvement data, since that is the method 
states are currently using, but the steps involved with planning for and conducting data collection, and analyzing and 
communicating your results, can be applied to other methods of data collection. If your state wants to use another 
method, keep in mind that it will be necessary to identify or develop valid and reliable data collection instruments and 
implement standard data collection and analysis procedures to ensure that the data collected are high quality. States 
interested in exploring other data collection methods can contact their IDC State Liaison for assistance or should 
consider working with a third-party contractor experienced in data collection. 


READY: How can my state ensure the data collected are 
representative? 


In the case of Indicator B8 data, “representativeness” refers to the extent to which the demographics of the students 
whose parents participated in data collection activities are representative of the demographics of all children receiving 
special education services in the state. Representativeness is very important. If your data are not representative, they 
will not paint an accurate picture of the nature and level of parent involvement in your state. For example, if only a 
small group of parents who are highly motivated to respond to your survey (such as those who have had very positive or 
very negative experiences related to parent involvement) participate in the survey, you may be over- or underestimating 
families’ feelings of involvement in their children’s special education programs. 


A number of factors influence whether and how well the demographics of the students whose parents participate in 
your data collection activities are representative of the demographics of all students with disabilities. Things to keep in 
mind and address are 


e Whether and how specific groups of parents are chosen (or not chosen) to participate (often referred to as 
“sample selection”) 
e Whether you have the information necessary to determine if the data you collect are representative 


e Whether your data collection instruments and procedures facilitate participation and allow you to track 


— The number of parents who participate compared to the number invited to participate (also known as 
“response rate”) 


— Whether parents who participate in data collection activities are similar to those who do not participate 
(which, in the case that they are not similar, might be an indication of “nonresponse bias”) 
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Carefully select the group that will participate in data collection 


To determine which group(s) your state will include in the Indicator B8 data 

SUS a cla ae What States Are 
collection activities, your state will need to look to its evaluation questions and 
the types of analyses it would like to conduct. For example, if the state wants Doing 
mete in-depth or Varies information about parent iON Ett they might Avproninarelnalbarenteseend 
involve school administrators, teachers, parents, and students in the data their Indicator B8 parent 
collection activities beyond those that are usually carried out for Indicator B8. involvement survey to a sample of 


parents rather than to all parents 
While it is technically possible to conduct a census of all parents of students with whose children receive special 
disabilities for your Indicator B8 data collection, it is often not feasible or education services (OSEP 2016). 
desirable. For example, surveying all the parents of students with disabilities in 
your state would be time-consuming, costly, and likely to result in a low response rate. Selecting a sample allows your 
state to save time and money and, when done correctly, can help to improve the quality and accuracy of your parent 
involvement data by leaving more time and resources for conducting ongoing checks of response rates and 
representativeness and following up with parents. 


When making decisions about whether and how to select a sample, consider the following: 


e How much time and money can be spent? Is it feasible to conduct a census? 


e How many people are available to work on the data collection and analysis activities (e.g., to recruit parent 
participants, collect data, analyze data)? 


e Will the analysis compare groups (e.g., compare the level of parental involvement in districts with certain types 
of special education programs to the level of involvement in districts without those programs)? If so, how many 
groups? 


e Does the study team plan to conduct descriptive statistical analysis, inferential statistical analysis, or qualitative 
analysis of interview/case study data? 


— What is the minimum sample size needed in order to reach conclusions with a pre-specified level of 
confidence (when conducting statistical analyses of the results)? 


Sampling is an area where a team member—or a third-party contractor—with specific training, expertise, or 
experience is needed, as there are multiple technical details that must be considered. For more information, consult 
the important information about sampling provided in the Resources section. We also recommend that you contact your 
IDC State Liaison to connect you with resources and individuals with expertise related to sampling procedures. 
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Gather data on demographics and other 


important considerations related to A Note About Keeping Data 


representativeness Anonymous 

When planning your state’s data collection activities, you must Keeping data collection anonymous can hinder your 
take steps to ensure you will have the information needed to efforts to collect high-quality data by, for example: 
track response rates ane assess FEPIESeDRATIVENESS of me data Saivonine denardentaniaelressoneeeteitiod 

you collect. When thinking about collecting demographic and whether the particular individuals you have asked to 
other data, it is important to know the difference between participate in your data collections actually 
collecting anonymous data and keeping data confidential. participate and to calculate response rates 


Anonymous data are data that cannot be traced to the parent or = ° Requiring that you ask your participants a lot more 

student, while confidential data can be traced but will not be questions to be able to report on 

improperly disclosed. We strongly recommend that you do not popes aw enesSy Nisbet ne eneth eh wile dake 
. collection instruments and potentially increasing the 

try to keep your parent involvement surveys (or other data 


. esa agp a burden on responders 
collections) anonymous, but it is still important to maintain the * Forcing you to send out reminders to the entire 


security (and confidentiality, if promised to responders) of your group of potential participants, rather than targeting 

data (see the notes about Keeping Data Anonymous and Data specific individuals to get them to respond, which 

Security and Confidentiality for more information). may annoy participants who have already 
responded 


A Note About Data Security and Confidentiality 


Data security is extremely important. Data protection processes and systems should be in place before data collection begins. 
Safeguards are imperative if parents were assured confidentiality. As a reminder, confidential data can be traced but will not be 
improperly disclosed. Here are some steps you can take to protect confidentiality as well as ensure the security of data: 


e Use ID codes on surveys in place of identifying information 

e Separate files that contain student or parent identifiers (e.g., names and addresses) from survey responses; the unique ID 

number can be used to link the files 

Encrypt personally identifiable information 

Limit access to the electronic and hard-copy data and track user access 

Use security codes and password protection to restrict access to computerized records 

Automatically lock users out of the electronic system after a certain period in the event someone forgets to log off 

e Install firewalls 

e Use secure data transfer protocols 

e Conduct regularly scheduled updates to antivirus software 

e Use locked data filing for hard copies of data collection instruments and forms 

e Properly dispose, destroy, or delete study data/documents, when appropriate, using secure methods of data destruction, 
including destroying backup copies, shredding paper copies, and electronic “wiping” of computer disks 


The recommendations we make in this toolkit assume that you will not try to keep your data collections anonymous, but 
that you will work hard to keep your data secure and maintain confidentiality, if promised. 


Below are some of the steps you can take to ensure you will have the information you need: 


e Send out a unique identifier for each survey (or other data collection instrument) so that you can track 
responses and match surveys to demographic information 


e Collect demographic and other data on 
— All parents of children with disabilities in the state or district 
— Parents who respond to the survey (also called “responders”) 


— Parents who do not respond to the survey (also called “nonresponders”) 
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For example, if you are going to be looking at the proportion of surveys received by race/ethnicity, geographic variables, 
child’s disability category, child’s gender, child’s age, length of time in services, income, and/or primary language, you 
will need to ensure that you have access to these data before you send your surveys out or you will need to collect the 
data through the survey itself. 


If demographic data are currently part of a separate data file or system, you will need to use a consistent linking variable 
across files, such as a unique ID number. You should create the linking variable in your demographic file and your survey 
file before you send out any surveys so you know that you are able to track the demographic information of responders 


(and nonresponders) when you begin receiving completed surveys. 


Choose or design data collection instruments 
with participation and representativeness in 
mind 


Whether you are choosing which existing data collection 
instrument(s) you will use to gather your Indicator B8 data 
(such as the NCSEAM scale) or designing a new instrument, 
keep in mind that the design of data collection instruments 
can affect whether and how individuals respond. A well- 
designed data collection instrument will make it easier for 
individuals to respond fully and accurately and be 
accessible to all potential respondents. As you plan for 
data collection, be sure to 


e Use or create instruments that facilitate 
participation 


e Make instruments accessible 


We have included two survey examples in the online 
toolkit that you can refer to if you intend to create a new 
survey. The purpose of the sample surveys is to 
demonstrate possible layouts and illustrate survey design 
principles described in this toolkit (see the section on 
modifying or creating a new survey for more information). 
Tips presented in this toolkit are noted throughout the 
sample surveys. The example items included in the sample 
surveys are ones commonly found in different surveys that 
states use. 


How Can Stakeholders Help? 


Pennsylvania developed its infant/toddler and preschool 
early intervention survey in tandem with the family survey 
process that spans Pennsylvania’s early childhood 
programs. These included the Pennsylvania Pre K Counts 
program, Head Start State Supplemental Program, Home 
Visiting programs, and Keystone STARS (TQRIS). The state 
wanted to develop a survey that would contain core items 
to use across all of its early childhood programs and 
maintain items that provide feedback on more specialized 
areas of the programs (e.g., items related to timely service 
delivery). The survey development process started with an 
internal stakeholder group, comprising staff from different 
programs in the Office of Child Development and Early 
Learning. This internal group reviewed a series of existing 
surveys and created an item bank from those. The internal 
stakeholder group then rated the items based on their 
relevance and developed a draft survey. To ease the 
transition to a new survey, the group maintained most 
items from the previously used NCSEAM survey. A broad 
array of stakeholder groups, including a family advisory 
group and the State Interagency Coordinating Council, 
then reviewed the survey and provided feedback that 
informed revisions. The state also engaged in a pilot of the 
survey and used feedback from the process to make final 
changes to the survey. 


Table 3 includes some issues that might arise if a state administers a poorly designed survey, information about the 


possible effect on survey respondent behavior and the quality and representativeness of data, along with some possible 


solutions. 
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Table 3. How instrument design can affect respondent behavior, data quality, and representativeness 


Instrument design issue 


The definition of “parent 
involvement” is unclear, so the 
interpretation of what constitutes 
“parent involvement’ is left up to 
individual responders. 


Possible effect on respondent behavior 


e Responders have different ideas of what 


“oarent involvement” means, so responses 
cannot be easily interpreted. 

Responses related to the level of 
involvement vary widely from one district 
to the next, even if the actual level of 
involvement is not that different (based on 
data collected through other sources). 
Responders choose not to answer the 
questions because they do not understand 
what is meant by “parent involvement.” 


Possible effect on data quality and 
representativeness 

Data collected through the survey are not 
reliable because the survey will produce 
different results depending on the 
respondent’s interpretation of “parent 
involvement.” 
Data collected through the survey are not 
reliable because the survey will produce 
different results depending on the 
respondent’s interpretation of the level of 
involvement. 
Data may not be representative if certain 
parents decide not to answer questions 
based on different understandings or 
because they are confused. 


Possible solution 
Include a clear definition of “parent 
involvement” at the beginning of the survey 
so all responders know how to interpret the 
term. 
Get stakeholder input on how to define 
“parent involvement.” 
Include clear definitions of the different 
levels of involvement (e.g., provide 
definitions of what low/ moderate/high 
involvement looks like) so that responders 
will respond consistently. 


Individual survey questions (also 
called “items”) are unclear or poorly 
written (e.g., include multiple ideas 
within one question). 


Parents do not know how to respond to the 
question so they do not answer it. 

Parents only answer one part of the 
question (for items that include multiple 
ideas). 


Data collected through the survey are not 
reliable because the survey will produce 
different results depending on the 
respondent's interpretation of individual 
questions. 

Data may not be representative if certain 
parents decide not to answer questions 
based on different understandings. 

Data may not be representative if certain 
parents decide to stop the survey because 
they get frustrated. 


Be sure that all survey items are well 
designed (see the section on how to modify 
or create a survey for more information on 
how to design high-quality items). 

Pilot test survey items with different 
potential respondent groups to see if they 
understand them in the same way. 


The survey does not include clear 
instructions for how the respondent is 
supposed to proceed from one section 
to the next (especially when using 
questions that “skip” responders to 
different sections based on how they 
respond to a particular item). 


Parents who are supposed to answer 
specific questions do not answer them 
because they went to the wrong section. 
Parents who are NOT supposed to answer 
specific questions answer them because 
they went to the wrong section. 

Parents get frustrated and do not complete 
the survey. 


Data collected through the survey are not 
valid because the responders did not 
answer the questions correctly. 

Data may not be representative if certain 
parents decide to stop the survey because 
they get frustrated. 


Use an online survey that has “skip 
patterns” built into it so that parents are 
automatically moved from one section to 
the next based on their responses. 

Pilot test the survey with different potential 
respondent groups to see if they 
understand how to complete it. 
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Instrument design issue 
The survey is only available in English 
and/or is not made accessible to 
individuals with disabilities. 


Possible effect on respondent behavior 
Parents who do not read or speak English 
cannot respond to the survey. 

Parents with disabilities cannot access the 
survey so they cannot respond. 


Possible effect on data quality and 
representativeness 
Data will not be representative if certain 
parents are unable to answer the survey 
because of a language barrier or another 
type of accessibility issue. 


Possible solution 
Offer the survey in various languages, 
based on the demographics of your 
state’s population. Be sure to use trained 
translators! 
Offer accessible versions of all surveys for 
individuals with disabilities. See the 
regulations for Section 508 compliance 
for more information on how to make 
instruments accessible. 
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Create data collection procedures and tools to facilitate participation and track responses 


Just as the design of the data collection instruments affects data quality and . 
ih hea salen What States Are Doing 

representativeness, the way you go about collecting data is also important. 

Here are some tips to help you increase participation and ensure that you are Almost all states use self-administered 

gathering high-quality, representative data. paper or online surveys to collect data 


on parent involvement. 


e Consider when and for how long you will collect data 
e = Think carefully about which mode of data collection to use 
e Be sure to have updated contact information for potential responders 
e Set up a database to track data collection and plan to do nonresponse follow-up 
Timing Is Important. Two elements of timing are important for data collection: scheduling and the field period. 
e Scheduling: Conducting data collection activities at inconvenient times may increase respondent burden, result 


in overlapping data collection efforts, and ultimately decrease response rates. Coordinate data collection efforts 
with state and local officials and avoid schoo! breaks and holidays whenever possible. 


e Field Period: Field period refers to the amount of time planned for data collection. Allow sufficient time for all 
data collection efforts, including 


— Time to send surveys to parents 
— Time for parents to complete and return the survey 


— Time for data collectors to follow up with parents who do not respond initially 


Mode Matters. Data collection mode or delivery method can affect both data quality and response rate. Common issues 
associated with different survey modes include 


e Online surveys 


— Internet access: Many households still lack internet access; those with internet access differ from those 
without internet access, with more affluent households and non-Hispanic White households having greater 
access.° Use more than one mode (e.g., online and paper) to minimize potential issues with 
representativeness and response bias. 


— Browser/mobile access: Parents may have difficulty responding to a survey using a browser that does not 
adequately support the survey.*° Additionally, many parents access the internet through their mobile 
devices. Design online surveys with these issues in mind. 


— Email filters: Many parents have email filters that block “unwanted” or unsolicited emails. Send surveys 
from a state agency or other official email address to make it more likely that surveys are delivered to their 
intended recipients. 


e Paper surveys 


— Survey complexity: Surveys with complicated question structures (e.g., those that filter how a respondent 
proceeds through questions based on responses to prior questions) can be difficult to complete when they 
are in paper format, which may affect responses and response rates. Create uncomplicated paper surveys, or 


° File and Ryan 2014. 
10 Tourangeau, Conrad, and Couper 2013. 
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use online surveys to program question patterns and make it more likely that responders will answer the 


questions they are intended to answer. 


e Online & paper surveys 


— Perceived importance: Many potential responders overlook emails or letters inviting them to participate in 
data collection because they are not aware of the importance of the effort. Use official letterhead in 
communications and enlist support of stakeholder groups to notify potential responders of the importance of 


participating in data collection. 


Contacts Are Critical. Having complete and accurate contact information is very 
important. The better the quality of parents’ contact information, the better 
the chance of achieving a higher response rate. 


e When planning for data collection 


— Enlist district and school staff support to review, verify, and update 
parent contact information. Obtain physical and email addresses 
and phone numbers to allow for nonresponse follow-up. 


— If using mail surveys, use an address correction service such as the 
ACS service from the U.S. Postal Service to request address 
corrections. Include a unique code in a specified format on the 
mailing label to receive electronic notice about any address 
changes. There are fees associated with this service based on the 
number of address changes. 


— Ask parents to verify contact information at IEP meetings or other 
meetings at school. 


e During data collection 


— If using online surveys, investigate any bounce-back emails for data 
entry errors or inactive accounts. Follow up on email bounce-backs 
by mail or by phone. 


Tracking Is Key. To monitor response rates and representativeness, you must 
have a system to track your data (such as Microsoft Access or Excel). See the 


section on creating a data file for more information about how to set up your 
tracking file to ensure that the data you enter are accurate. 


e Unique IDs: The only way to know who responds to your survey is to 
include a unique ID on each survey. Some online services do this 
automatically, whereas paper surveys will require the state to add the 


What States Are Doing 


Connecticut modified its survey 
distribution process in an effort to 
increase response rates. The state 
sent districts an Excel template with 
the state-assigned student identifiers 
for the special education students in 
their district and asked districts to 
enter the most current mailing 
address for each student. This process 
was a change from prior years when 
the state sent districts an Excel 
spreadsheet with students’ mailing 
addresses prepopulated (as extracted 
from the state’s special education 
data system) to confirm the mailing 
addresses. This revised process helped 
to ensure that each district reviewed 
and edited the Excel spreadsheet with 
the most current addresses. In 
addition, districts were able to 
indicate if a child had moved out of 
the district or if any additional 
students had been identified. The 
state also asked districts to provide 
parents’ email addresses, when 
available, to allow for direct parent 
access to the survey through a 
personalized link. Nearly all districts 
were able to provide emails for some 
or all parents of students with 
disabilities. 


ID number to each survey. Ensure all surveys include a unique ID for tracking purposes. 


e Database: Many online survey services automatically combine survey tracking and response monitoring ina 
single database, allowing you to monitor response rates in real time. With these services, survey responses are 
automatically entered in digital format, reducing the amount of work needed later to enter data. Design your 
tracking system to track surveys that were sent out and those that were received. 


e Response rates: There are two primary types of response rates to consider: overall response rates and item 
response rates. Low response rates can affect data quality and can be especially problematic if certain groups of 
parents are more (or less) likely to respond than others. Combine information about response rates with 
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information about the demographic characteristics of your parent population to learn more about the 
representativeness of your data. 


— Overall response rates: The proportion of completed surveys compared to the number of surveys sent out. 
Calculate overall response rates to know whether the group of responders is representative of your overall 
parent population. 


— Item response rate: The proportion of parents who responded to a particular question compared to the 
total number of parents who responded to the survey. Parents might accidentally skip a particular question 
or deliberately skip one that is hard to understand or that they do not want to (or cannot) answer. Calculate 
item response rates to know whether particular groups of parents were more or less likely to respond to 
certain items on the survey, which may affect the representativeness of your data. 


e Nonresponse bias: Nonresponse bias occurs when the individuals who complete the survey differ in meaningful 
ways from those who do not. For example, if parents who are less involved in their children’s education are less 
likely to respond than parents who are more involved, the results of the survey will be biased and parents who 
are less involved will be underrepresented. Plan to conduct nonresponse follow-up to minimize the possibility of 
bias and take steps to check for nonresponse bias after data are collected. 


Carefully select the mechanism for getting parents to participate 


Once you have notified parents (or school administrators, teachers, or other groups you may be collecting data from) 
about the data collection effort, you need to make sure they follow through and actually participate. You might do this 
by 

e Sending the survey directly to parents by mail or email 


e Giving the paper survey to parents during in-person meetings or at pick-up/drop-off times for them to complete 
at a different time and return to the school 


e Having parents complete the survey when they visit the school 
e Sending the survey home with students for parents to complete and return to the school 


Table 4 presents different approaches your state might adopt to deliver the survey to parents, along with benefits, 
drawbacks, and considerations for each approach. No one approach is the best, so think carefully about your parent 
population and consider how you can get as many of them as possible to participate. Consult with district/school staff 
and other key stakeholders for suggested approaches and support. Consider a rolling approach: 

e Start with one approach first, such as emailing parents a link to an online survey 


e Give parents time to respond 


e Follow up using another approach, such as mailing out links to the survey or hard-copy surveys to parents who 
have not responded 


In the section on working to improve response rates, you will see some sample timelines for mail and online surveys that 
you can use to guide your efforts. 
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Table 4. Approaches to delivering surveys to parents 


Approach 
Mail 


Email 


In-person 
during 
meetings, 
pick-up/ 
drop-off 
times 


Good to use when... 
Only mailing 
addresses are 
available 

High percentages of 
parents have no or 
limited internet 
access 


Complete and 
accurate email 
addresses are 
available 

Parents already 
expect to receive 
school/ district 
communications via 
email 


Most parents 
regularly visit the 
school for meetings, 
including IEP 
meetings, pick- 
up/drop-off 

Parents do not have 
physical mailing 
addresses (e.g., 
transient populations) 
and email addresses 
are unavailable or 
parents have no or 
limited email access 


Benefits 
Familiar approach—parents 
often expect mail 
communications from schools 
and districts 
Allows for sending paper 
surveys or links to online 
surveys 


Inexpensive—no need to pay 
for staff time to address and 
mail materials; no postage 
costs 

Allows for embedding a link 
that takes parents directly to 
the online survey 

Online survey services allow 
for tracking who receives and 
completes the survey 
Ensures parents who visit the 
school receive the materials 
Limits need for follow-up to 
mailings/emails 

Allows parents who do not 
have physical mailing 
addresses or email addresses 
to participate if they visit the 
school 

Inexpensive—no need to pay 
for staff time to address and 
mail materials; no postage 
costs 


Drawbacks 
Requires staff time to prepare and mail 
materials; postage costs 
Follow-up will likely be required, which 
increases cost 
No way to monitor whether the mail is 
opened or whether it is opened in time for 
parents to respond within the data 
collection period 
The person who opens the mail may not be 
willing or able to complete the survey (e.g., 
because she is not the parent, or because 


he does not know the answers) 


Spam filters may block emails 


Parents may quickly shift their attention to 
other incoming emails 

Parents may be reluctant to click on 
embedded links due to fear of malicious 
software or phishing scams 

Follow-up will likely be required, which 
increases staff time 


Only parents who visit the school receive 
the materials, limiting the number of 
parents who can participate and possibly 
reducing the representativeness of the 
data 

Surveys might get mixed up with other 
materials distributed during the meetings 
and overlooked 

Parents may be processing information 
discussed during the meetings and forget 
about the survey or consider it a low 
priority 

Parents may not pay careful attention to 
materials handed out during pick-up/drop- 
off due to other demands on their 
attention 


Keep in mind... 
Mailed materials should 
Look official 
Prominently display the sponsor(s) 
Clearly communicate the 
importance of the data collection 
Be easily understood 
Design survey materials to stand out 
(e.g., putting them in bright orange 
envelopes) so parents pay attention 
to them 
Include self-addressed stamped 
envelopes with the materials to 
encourage parents to respond 
Prevent emails from going to 
parents’ spam or junk folders by 
Using plain text rather than HTML 
messages 
Sending individual emails rather 
than using bulk mailing options 
Testing the email message before 
sending, using a spam analyzer 


Design survey materials to stand out 
(e.g., putting them in bright orange 
envelopes) so parents pay attention 
to them 

Separate survey materials from 
other materials that are distributed 
during the meetings or pick- 
up/drop-off 

Include self-addressed stamped 
envelopes with the materials to 
encourage parents to respond 
School staff must be able to answer 
questions about the survey, or 
provide contact information to 
parents with questions 
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Approach 
Survey 
access at 
school 


Send home 
with 
students 


Good to use when... 
Most parents 
regularly visit the 
school for meetings, 
including IEP 
meetings, pick- 
up/drop-off 

Parents do not have 


physical mailing 
addresses (e.g., 
transient populations) 
and email addresses 
are unavailable or 
parents have no or 
limited email access 


Parents already 
expect to receive 
school/ district 
communications in 
this way 

Parents do not have 
physical mailing 
addresses (e.g., 
transient populations) 
and email addresses 
are unavailable or 
parents have no or 
limited email access 


Benefits 
Ensures parents who agree to 
participate in this way 
complete the survey 
Limits need for follow-up to 
mailings/emails 
Inexpensive—no need to pay 
for staff time to address and 
mail materials; no postage 
costs 
Gives parents who have email 
addresses but no or limited 
internet access an opportunity 
to complete online surveys 


Inexpensive—no need to pay 
for staff time to address and 
mail materials; no postage 
costs 


Drawbacks 
Only parents who visit the school can 
complete the survey, limiting the number 
of parents who participate and possibly 
reducing the representativeness of the 
data 
Parents must have time and be willing to 
complete the survey at school, during 
school hours 
Additional staff and computer resources 
may be needed for parents to complete 
surveys 
Parents may feel pressured to respond 
differently (e.g., more positively) when 
they complete it on the school premises, 
which affects data quality 
Requires that students actually deliver the 
surveys to parents 
No way to monitor whether the materials 
are delivered or whether they are opened 
in time for parents to respond within the 
data collection period 
The survey may get mixed up with other 
school materials and be lost or forgotten 
Students or parents may not pay careful 
attention to materials due to other 
demands on their attention 
School staff will need to track survey 
responses and follow up with students 
whose parents have not completed the 
survey 


Keep in mind... 
More effort may be required to get 
parents to come to the school and 
complete the survey 
Keep the survey short to encourage 
parents to complete it while at 
school 
School staff must be able to answer 
questions about the survey, or 
provide contact information to 
parents with questions 


Design survey materials to stand out 
(e.g., putting them in bright orange 
envelopes) so parents pay attention 
to them 

Separate survey materials from 
other materials that are distributed 
during the meetings or pick- 
up/drop-off 

Include self-addressed stamped 
envelopes with the materials to 
encourage parents to respond 
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RUN: How can my state maximize participation in its 
parent involvement data collection activities? 


Carefully planning your state’s Indicator B8 data collection activities is an important first step in obtaining high-quality, 
representative data. As you transition to collecting data, certain actions will help increase the likelihood of a successful 
data collection effort. 


Be clear about who is sponsoring data collection 


Data collection activities supported or sponsored by organizations that parents recognize and consider legitimate may 
help improve response rates. Sponsorship conveys to parents a sense of purpose and indicates that the information they 
provide will be used to benefit them and their children. Whether your state is managing data collection activities 
directly—or working with stakeholders or third-party contractors to collect data—it is important to highlight the 
sponsor of the survey (e.g., state, district, and/or school) to demonstrate the authority and legitimacy of the effort. 
Because parents commonly receive communications from their child’s school and district, it is often a good idea to have 
one of them sponsor the effort, such as having a cover letter signed by a school or district official whose name parents 
may recognize. Prominently display the sponsor’s name and logo on all data collection materials. 


Formally invite potential responders to participate 


No matter how your state chooses to collect its Indicator B8 data (e.g., through 


a survey, interviews, or focus groups), you should formally invite potential What States Are Doing 

parents to participate. Whenever possible, personalize communications States reported a number of different 

(including envelopes, letters, and emails) by using the parents’ names, as methods for getting the survey to 

research has shown that doing so can boost response rates." parents, including mailing a paper 
survey, mailing an invitation to 

When inviting parents to participate, be sure to include participate by web, or emailing a link 
to the web survey when email 

e Abrief introduction that states the purpose of the survey and who will addresses were available. 


be collecting the data 
e How and why the parent was selected 
e Benefits to the parent (and/or the child) and to the community or state 
e Information about confidentiality or anonymity of the data 
e Explanation of how data will be used and kept secure 
e Assurance that participation is voluntary 
e Information about incentives (if you offer them) 


e Contact information for help with questions [Important note: Be sure to identify speakers of languages other 
than English in the schools or districts who can answer questions or provide needed assistance in the parents’ 
native language.’ Also include accessibility information for individuals with disabilities. ] 


e Atime estimate for completing the survey 


e Instructions on how and when to participate in data collection (e.g., how and when to submit the survey) 


4 Dillman, Smyth, and Christian 2014. 
12 Harkness, Villar, and Edwards 2010. 
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e Anexpression of your appreciation for the parent’s participation? 
Consider offering incentives 


Offering an incentive to potential participants can improve response rates, and there has been little indication that using 
incentives decreases the quality of the data collected. Additionally, although incentives add a cost to survey 
administration, they can reduce cost in other areas, such as reducing the need for extensive nonresponse follow-up. 


Here are some additional factors to consider when thinking about using incentives: 


e Monetary incentives are more effective than nonmonetary ones (e.g., pens, magnets) * 


e Prepaid incentives (i.e., incentives that are paid before the survey is completed) are more effective than 
promised incentives (i.e., incentives that are contingent upon survey completion) *® 


e Incentives might help improve representativeness of the data by encouraging parents who may not have 
otherwise completed the survey to do so 


e Consider offering incentives to districts and schools, such as recognition on the state website or opportunities 
for additional professional development, to enlist their support in encouraging participation in your data 
collection effort 


A Note on Using Incentives 

It is important to review district, state, and federal policies regarding incentives, as there may be specific guidance or prohibitions 
against their use. Some view offering incentives as an inefficient (or inappropriate) use of funds (i.e., taxpayers’ money). Using 
donations received outside of federal or state funding to support incentives can offset this concern. 


Monitor response rates 


As you begin collecting your parent involvement data, you should pay careful attention to response rates. The section on 
creating data collection procedures and tools to facilitate participation pointed out that there are two primary types of 
response rates: overall response rates and item response rates. Response rates are important because they give 
information about how well the data you collect represent the views of all of the parents of students with disabilities in 
your state. Table 5 defines these response rates and provides information about how they can inform data quality and 
representativeness. 


13 Adapted from Carlson and D’Agostino 2015. 
14 Singer and Ye 2013. 

15 Mercer et al. 2015; Singer and Ye 2013. 

16 Mercer et al. 2015; Singer and Ye 2013. 
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Table 5. Information about overall and item response rates 


What is it? How can it inform data quality and representativeness? 
Overall Overall response A high overall response rate can be an indication of data quality, IF 
response | rates tell youhow | e The survey is well-designed; AND 
rate many parents e The views expressed by the group of parents who respond are representative of the views of 
responded all parents of students with disabilities; AND 
compared to the e There is little missing data; AND 
total number of e There is no indication of nonresponse bias. 
parents who were 
invited to 
participate in data 
collection. 


A low overall response rate indicates that the data may not reflect how the majority of parents 
would respond. 


Given the same caveats listed for overall response rates, a high item response rate can be an 


Item Item response 
response | rates tell you indication of data quality. 
rate whether parents 


A low item response rate may be an indication of 
e Issues with survey design, such as 
— Unclear instructions for completing the survey and proceeding from one question to the 


were more likely 
to answer some 
questions in the 


survey than other Rex 
questions. — Confusing language that frustrates parents or makes it hard for them to understand how 
to respond 


— Sensitive language that makes some parents feel uncomfortable responding 
— Insensitive or inflammatory language that may cause some parents to refuse to respond 
e Nonresponse bias 


A Note on Nonresponse Bias 

Nonresponse bias occurs when certain groups of parents are more or less likely to respond to the survey or to certain questions 
within the survey. It is a particular concern for data quality and representativeness if demographic data show that the students 
whose parents responded to your survey (or to particular items) differ substantially from the overall group of students with 
disabilities. This may mean that certain groups who may have unique opinions or experiences may not be represented in survey 
responses and, in turn, in results for Indicator B8. In such instances, any decisions made that are based on survey results may not 
appropriately address parents’ and students’ needs. 
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Table 6 presents the equations for calculating overall and item response rates if you are using a census or a sample, and 
gives examples of each. Box 2 lists some online response rate calculators that you can use with your data. Do not wait 
until data collection is over to calculate response rates, because then it will be too late! As you will see in the next 
section, your state should take steps to improve response rates during data collection. 


Table 6. How to calculate overall and item response rates 


Overall Response Rate Example 


6,000 
10,000 


x 100 


# of parents who responded 
# of parents of SWD in the state 


Census x 100 


0.6 x 100 = 60% 


Sa 
500 * 


# of parents who responded 


Sample x 100 


# of parents in the sample 
0.55 x 100 =55% 


Item Response Rate Example (For the sample cited above) 


125 99 
as 


# of parents who responded to the item 


Census & Sample x 100 


# of parents who responded to the survey 


0.45 x 100 = 45% 


NOTE: SWD = students with disabilities. 


Box 2. Online response rate calculators 


AAPOR’s Standard Definitions is a resource for determining and assigning disposition codes and calculating 
response rates. 


National Post-School Outcomes Center Response Calculator, Instructions for the National Post-School Outcomes 


Center Response Calculator, and Post-School Outcomes: Response Rates and Nonresponse Bias paper are 
resources developed for Indicator 14, but can also be used for Indicator B8, to understand response rates and 


representativeness and to calculate an overall response rate by demographic variables. 


ECTA’s Response Rate and Representativeness Calculator is an Excel-based calculator developed to guide 
response rate calculations as well as representativeness for Part C family outcomes, but it can also be tailored for 


use with Indicator B8. 


What States Are Doing 


Rhode Island uses a live dashboard, provided by a third-party contractor, for state staff and district leadership to monitor 
districts’ and schools’ response rates. 
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RUN: What steps should my state take to improve data 
quality during data collection? 


Earlier you learned that response rates can be indicators of data quality when certain criteria about the data collection 
activities are met (e.g., the surveys well-designed, the group of parents who responded to the survey is representative of 
all parents of students with disabilities). In this section you will learn how to take certain steps to monitor data quality 
during data collection, including 


e Working to improve response rates based on what you learn from your response rate calculations 


e Checking representativeness of data using demographic data collected through the survey (or some other 
means) 


e Working to improve representativeness of data 
Additionally, you will learn how to plan to improve representativeness of future data collections based on the data you 
collect. 
Work to improve response rates 
You must monitor and take any necessary steps to improve response rates while data collection is ongoing if you 
want any chance of collecting high-quality, representative data. Here are some tips to help improve response rates 
during data collection: 

e Investigate incorrect contact information immediately 

e Send follow-ups and reminders 

e Track responses and follow up with those who have not responded (also called “nonresponders”) 


Investigate incorrect contact information 


No matter how well you plan, once you start data collection you will undoubtedly find errors in the contact information 
you have for parents. When this happens—and it will—it is important that you take immediate steps to obtain the 
correct information. You might do this by 

e Asking the district or school to verify the information 

e Sending a request for updated information home with students 

e Getting help from a Parent Center to contact parents and get the correct address/email/phone number 


e Asking parents to verify their contact information during conferences, IEP meetings, or other in-person 
interactions with parents 


e Calling the parent, or mailing a letter requesting the correct email—if using an online survey with email 
notifications 


The sooner you get the correct information, the sooner you can follow up and encourage the parent to respond, so plan 
to have staff time available for doing this task during the data collection period. 
Send follow-ups and reminders 


An effective strategy for increasing response rates is sending periodic follow-ups and reminders about the data 
collection activities and timeline to parents. Whenever possible, try not to send reminders to individuals who have 
already completed the survey. Table 7 and Table 8 give examples of processes and timelines for contacting parents for 
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mail and online surveys, respectively. When sending reminders or replacement mailings/emails, be sure to change the 
message, usually in the cover letter or email content, to remind parents of previous attempts to solicit a response. 


Table 7. Example of a process and timeline for contacting parents for a mail survey 


Contact Activity Timing 


1 week after initial 
mailing 

3 weeks after 
initial mailing 
6 weeks after 
initial mailing 


2nd replacement survey mailing (survey, 2nd replacement cover letter, & return envelope) 
sent via FedEx or USPS Priority Mail 
Source: Adapted from Dillman, Smyth, and Christian 2014. 


Table 8. Example of a process and timeline for contacting parents for an online 
survey, with email as primary method of contact 


Contact Activity Timing 


5 days after mail prenotification 
3 days after web survey invitation sent 


2 weeks after web survey invitation sent 


3rd email reminder 1 month after web survey invitation sent 


Source: Adapted from Dillman, Smyth, and Christian 2014. 


The examples of processes and timelines presented here are general guidelines; if desired, you may want to adjust the 
timeline by 


e Increasing timing between contacts for mail surveys to account for possible delays in mail delivery 
e Reducing timing between contacts for online surveys because parents are not required to mail anything back 


Track responses and follow up with nonresponders 


In the section on creating data collection procedures and tools to facilitate participation, we discussed the importance of 
tracking responses—an4d, in particular, the importance of having a unique ID for each survey. As you start collecting 
data, the database or other data tracking system you created while planning your data collection will become very 
useful. 


If you are doing an online survey, you may be able to download data in real time to identify which parents have 
completed the survey out of all of the surveys that were sent. You also can track who accesses or starts the survey but 
fails to complete it, suggesting a parent may be interested in participating but has not yet done so. This might be an 
indication of an issue with the survey, or just a result of the parent getting interrupted or distracted. Either way, it is 
important to follow up with those parents to encourage them to complete the survey. If they will not complete the 
survey, try to obtain their reason for not completing it, which may give you information about issues with survey design, 
nonresponse bias, or some other issue. 


If doing a mail survey, you will need to enter the data into your data tracking system. You should have already created 
procedures for entering data when planning your data collection. If not, go to the section on creating data collection 
procedures and tools to facilitate participation to learn more about it. It might be possible to scan the surveys into a 
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digital format—if your survey has this capability; otherwise, you must enter the data manually. Here are some things you 
need to do as you collect your data: 


e Create clear and consistent disposition codes for the various survey statuses (e.g., complete, unreturned, 
refusal, ineligible, nonlocatable). 


e As you receive surveys or learn information about the parent, enter and update the codes for each parent in the 
tracking system. 


— Use the unique survey ID to track the survey status for each survey that was sent out. You may also want to 
create a field that indicates the date that the tracking status was entered or updated. 


— Follow up with parents that have not yet responded during the period of data collection. Try different 
methods of contacting parents (if you have different types of contact information, to increase the likelihood 
that you will reach the parent). 


— Clearly mark the final disposition code for each survey. This is the information that you will use to calculate 
your overall response rate and assess the data for representativeness later on. 
Conduct periodic checks for representativeness 


Using demographic data collected from the parents who respond to your survey, you should conduct periodic checks for 
representativeness. You might first do this after you have received responses from approximately one-third of the 
expected number of participants and then again after you have received responses from about two-thirds of expected 
participants. In this section you will learn about 

e Determining whether data are representative 

e Comparing the characteristics of students whose parents complete the survey with those of all students with 

disabilities 

e Comparing the characteristics of responders and nonresponders 
Decide how you will determine whether data are representative 
There is no one “threshold” for determining whether data are representative. In fact, different organizations have set 
different thresholds for what constitutes “acceptable differences” among groups. For example, 

e The National Post-School Outcomes Center uses a threshold of +3 percent 


e The Early Childhood Technical Assistance (ECTA) Center’s Response Rate and Representativeness Calculator uses 
tests of statistically significant differences 


As you consider the representativeness of your data, we recommend that you consult with key stakeholders, such as 
Parent Centers, for input and guidance. You can also talk with your IDC State Liaison for assistance. 


Compare characteristics of students whose parents completed the survey with all students with disabilities 


One way to test for representativeness is to compare the demographic characteristics of students whose parents 
responded to the survey to the characteristics of all students with disabilities in the state or district. *” In fact, OSEP 
requires states to analyze the extent to which the demographics of the students whose parents responded are 
representative of the demographics of children receiving special education services. States should consider categories, 
such as 


e Race/ethnicity 


V Garrison-Mogren 2007. 
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e Sex 

e Age of the student 

e Disability category 

e Geographic location in the state*® 


When examining representativeness, ask yourself this question: “Are some groups under- or overrepresented among 
survey responders?” Or, to put it another way, you can ask “Were parents with certain characteristics more likely to 
respond to the survey than others?” ? Consider Figure 1, which presents data for the students whose parents responded 
to a survey with all students with disabilities in a fictitious state. What do the data show? 


Figure 1. Comparing students whose parents responded to all students with 
disabilities in the state 


Two or More Races i ee v 
Native Hawaiian or Other Pacific Islander | oe v 
Black or African American Pm 38% 
Asian 5 


American Indian or Alaska Native 


: . , 2%, 
Hispanic/ Latino 2% 10% 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Student rac/ethnicity 


Percent of students 


m@ Students whose parents responded BAI! students with disabilities 


NOTE: The numbers in the figure may not add up to 100 due to rounding. 


Clearly, the survey data presented in Figure 1 are not representative at this point. Specifically, White students are highly 
overrepresented, while Black or African American and Hispanic/Latino students are underrepresented, relative to all 
students with disabilities. 


Compare characteristics of responders and nonresponders 


Another way to test for representativeness is to compare the students whose parents responded (“responders”) with 
the students whose parents did not respond (“nonresponders”). Using the demographic characteristics of both groups, 
you can compare them and see if they differ, as in Figure 2. 


18 U.S. Department of Education 2014. 
19 Lammert, Heinemeier, Schaaf et al. 2016, p. 74. 
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Figure 2. Comparing responders to nonresponders 


Two or More Races 0 356 v 
vic TT § 2°; 
Native Hawaiian or Other Pacific Islander ore Vv 
Black or African American PO 62 A3% 
Asian me 3% 


American Indian or Alaska Native | ie "4 


Hispanic/ Latino | 15% 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 
Percent of parents 


Student race/ethnicity 


mResponders mi Nonresponders 


Here, too, the data collected so far are not representative. Specifically, compared to nonresponders, responder parents 
of White students are highly overrepresented, while parents of Black or African American and Hispanic/Latino students 
are underrepresented. Both Figure 1 and Figure 2 indicate that if you do not take steps to improve the 
representativeness of your data prior to the end of the data collection period, the data you collect will not accurately 
represent the views of all parents of students with disabilities in the state. 


Work to improve representativeness 


Once you have looked at the representativeness of the initial data, you will need to work to address any issues as quickly 


as possible. If you were careful from the start to select potential participants who would be representative, a 
straightforward way to address any issues is to make an extra effort to improve response rates, because that will 


automatically improve representativeness. Other, more targeted ways to improve representativeness during data 
collection might involve 


e Sending additional reminders and follow-ups to parents who have the specific characteristics that are 
underrepresented 


e _Enlisting support from Parent Centers or other key stakeholder groups to encourage parents from 
underrepresented groups to respond 


e Calling parents who started but did not complete the survey to encourage them to respond either in the original 
format (e.g., online or mail) or during the phone call 


In the next section you will learn how to reflect on what you can learn from the data. This process of reflection involves 


e Preparing the data for analysis so that you can be sure you are working with high-quality data 
e Conducting rigorous analyses that will enable you to have confidence in your findings 
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REFLECT: How can we prepare our data for analysis? 


Regardless of the way you collected your data, it is important to first check the data for accuracy and completeness. You 
should ensure that the data are as close to perfect as possible and ready for analysis. In this section you will learn how to 
e Create a data file 
e Enter the data accurately and completely 
e Clean and code the data 
Create a data file 


You should set up a data file to manage all data, preferably before data collection begins. Your data file can be part of 
the same database that you are using to track responses, or it can be a separate file. See the section on creating data 
collection procedures and tools for some important considerations about data security. Also, consider using the tips 


listed here when you prepare for data collection. 


Regardless of how you choose to organize the two files, your data file will ultimately contain all of the responses that 
parents provided on their surveys. Right from the start, take some time to consider 

e Which software program to use to manage quantitative and qualitative data 

e How the data file will be organized 


If you administered paper surveys and are entering the data by hand or scanning the data, you may choose to use a 
spreadsheet program like Microsoft Excel or a statistical software program like SPSS or SAS. Here are several tips for 
setting up the data file:° 


e Tip 1: Each row should include all of the data for one parent (each with a unique ID number), and each column 
should correspond to a different variable of analysis. Variables might include survey items, respondent 
demographic information, district or school information, etc. See Table 9 for an example. 


Table 9. Example of organizing data in rows and columns in a data file 


NOTE: Q1 = survey item 1, Q2 = survey item 2, etc. 
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e Tip 2: Create a table with information about each variable in the data file, including 


— The variable name (which may be the survey item number or an abbreviated label that describes the 


variable) 


— Adescription for each variable, the unit of measurement, and the type of data (e.g., text, numeric, date) 


— The maximum size of the entry (i.e., how many characters can be typed into the file) 


— The possible values or any value restrictions 


If you are working in a program like SPSS, the program will automatically direct you to do this; if you are using a 
program like Microsoft Excel, you may want to create this table in a new worksheet or in a separate file. See 
Table 10 for an example that illustrates a variable information table corresponding to the example in Table 9. 


Table 10. Example of table with information about each variable included in the 


data file 


WETatle) ( 


name Description 


SEX 


RACE 


Qi Survey Item 1: [DESCRIPTION OF 
ITEM] 


Data 


Units type MaxSize 


a ee a 


Race/ethnicity 


i ; 


_ —_— ie 


numeric 


Values and value restrictions (blank=none) 
3-21 


0 = Female 
1=Male 


1 = American Indian or Alaska Native 

2 = Asian 

3 = Black or African American 

4 = Hispanic/Latino 

5 = Native Hawaiian or Other Pacific Islander 
6 = White 

7 = Two or more races 

dd/mm/yyyy 


1 = Very strongly disagree 
2 = Strongly disagree 

3 = Disagree 

4 = Agree 

5 = Strongly agree 

6 = Very strongly agree 
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Tip 3: Set restrictions on cells in the data file to minimize data entry errors. For example, you can set up cells to 
accept only text, numbers, or dates. Also, you can limit the number of characters allowed in a cell and specify 
the permissible range (e.g., only allow ages 3 to 21 for student’s age). See Figure 3 for an example of how to do 


this in Excel. 
Figure 3. Example of setting age restrictions on cells in Excel data files 


Home Insert Page Layout Formulas View Q Tell me what you w 
—— Fal Show Queries [2] Connections A Z1A 
| ee | <> ‘ees Z¥ |AIZ 
Y ~ FS From Table [z=] Properties 
From From From From Other Existing New — Refresh _ Z| Sort 
Access Web Text Sources Connections | Query L@ Recent Sources All =» Edit Links 


Get & Transform Connections 


Get External Data 


te 


Data Validation 


Settings | Input Message | Error Alert 


Validation criteria 
Allow: 
Whole number 


ix] Ignore blank 


Data: 
between [| 
Minimum: 

3 
Maximum: 
21| 


al Apply these changes to all other cells with the same settings 


Clear All 
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Figure 4 presents an example of the error message that you will see if you try to enter an age outside of the prespecified 
range (as established in Figure 3). 


Figure 4. Example of entering incorrect values into the database 


Home Insert Page Layout Formulas View Q Tell me what 
fe a. ZZ [ Fr =| = Fal Show Queries Ts, LZ! Connections Al 
: BB <> = 
a Sten ; FA From Table (:] Properties 
From From From From Other Existing New — Refresh — | 
Access Web Text Sources Connections | Query L@ Recent Sources All » Edit Links i 
Get External Data Get & Transform Connections 
* fe | 22 
4 A B (= D E F G H I 
4 | | | i | | | 
1 \ID AGE SEX RACE DATE Ql Q2 Q3 a4 Qs 
mA 
2 |00001 22 
=A 
3 |00002 
¥. 
4 |00003 
— 
5 |00004 
6 | Microsoft Excel 
7 
8 | This value doesn't match the data validation restrictions defined for this cell. 
af 
al 
11 
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e Tip 4: When possible, create drop-down menus to help ensure consistency of data entry. The drop-down menu 
will only allow users to select preset responses. Figure 5 and Figure 6 illustrate this. A simple Google search of 
how to create drop-down menus in Excel will produce step-by-step instructions. 


Figure 5. Part 1 of creating drop-down menus in Excel 


Set Up Data File.xlsx - Excel 


Q Tellm 


Review View 


Insert Page Layout Formulas 


Home 


ES . =] Show Queries s, L2] Connections a [ZTA) ww ¥clear 
jaca } Z 
Get External = New . =F aaaiaes Refresh El oe Sort Filter Ye Reapply 
Datay §Query~ | (@ Recent Sources All |< Edit Links Ve Advanced 
Get & Transform Connections Sort & Filter 

Al ¥ Se 

A| A | B | L& | D | E | F | G | 
1 |Very strongly disagree 

2 |strongly disagree 

3 | Disagree 

4 “| Agree 

5 | Strongly agree Validation criteria 

6 | Very strongly agree Allow: 

7 List 

8 Data: 

9 between 

10 Source: 

11 =Sheet2!SAS1:SAS6 

12 

13 

14 |_| Apply these changes to all other cells with the same settings 

15, 

7 
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Figure 6. Part 2 of creating drop-down menus in Excel 


ee Ie Set Up Dat 
Home Insert Page Layout Formulas Review 
b > Fal Show Queries , [2)Connections a; PTR 
B) | 7B; le il 
- EE] From Table Properties 
Set External New Refresh Z| Sort | 
Data ~ Query ~ [% Recent Sources All~ Edit Links A 
Get & Transform Connections Sor 
F2 a fe 
| A | B | is | D | E E G 
1 |ID AGE SEX RACE DATE 
2 [0001 
3 = 
4 00003 
iA 
5 |00004 
6 
6 = — 


e Tip 5: Set up a separate data file for responses to open-ended items to make the analysis of qualitative data 
easier. You can use a spreadsheet, such as Excel, or a software program designed specifically for analyzing 
qualitative data, such as NVivo, ATLAS.ti, or Dedoose. See the section on analyzing qualitative data for more 
information about how to analyze open-ended responses. 


Online survey programs often allow you to automatically export the data from the survey into your data file. There are 
usually instructions available within the online survey program that describe the steps for exporting the data, and in 
many cases, you will have the option to export the data to Excel, SPSS, or SAS. You can take some steps when creating 
your web survey to ensure that your data file will be set up correctly when you download the data. For instance, 7? 


e Consider designing your web survey with checkboxes or radio buttons to restrict the way participants enter 
responses and to help ensure your data are clean and consistent when you download them, as illustrated in 
Figure 7. 


Figure 7. Example of using radio buttons to restrict responses 


2. What is your child's gender? RJ 


Female 
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e Verify that all items have the correct format so that, for example, a parent can check more than one box if a 
question asks responders to “select all that apply,” as shown in Figure 8. 


Figure 8. Example of using the correct format for an online survey question 


1. Which race/ethnicity best describes your child? (Check all that apply.) {J 


American Indian or Alaska Native 
Asian 

Black or African American 
Hispanic/Latino 


awaiian or Other Pacific Islander 


Each online survey program has its own processes and procedures for setting up the survey, so we recommend that you 
consult someone with expertise in online survey development to ensure the survey is set up correctly. You can also ask 
your IDC State Liaison for assistance. 


Enter data 

: , os a tla 5 
As you begin to collect data, be sure to track responses in your tracking file! What States Are Doing 
Then you can start entering the data into your data file (which might be the 
same as, or separate from the tracking file). The process you use will depend Arkansas, Florida, and Rhode Island are 
on what format you are using to collect your data (e.g., paper or online some of the states that have used 
survey) scannable forms to collect their survey 


data. This helps to minimize data entry 
; ; . ; : errors and helps ensure data consistency. 
e With online surveys, responses typically are entered directly into a 


database by the online survey program as parents complete the survey 
and then can be downloaded and imported into a data file for later analysis. 


e With paper surveys, you can choose to scan in results (if the surveys are on a scannable form) or enter the data 
by hand. If you decide to scan your surveys, be sure to train staff on the scanning equipment and software. 


If you need to enter the responses by hand, you must train staff on how to enter data accurately. Establish standardized 
data entry procedures and document the instructions that staff need to follow. 
e Explain exactly how data should be entered (refer to Table 10 for an example). 


e Provide a set of instructions with the specific data entry requirements and include examples of correct and 
incorrect entries. If possible, include the instructions within the data entry or spreadsheet file so that staff can 
refer back as needed. 


e Stress to staff that they should save and back up the data often.” 
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Even with a carefully trained data entry staff, you should take measures to ensure that staff are entering data correctly 
and completely. It is important in this step of the process to ensure that staff are entering the responses into the data 
file as they are received. It is not the place of the data entry staff to try to clean the data at this point—that will come 
later. Table 11 outlines some steps you might take. 


Table 11. Steps you might take to ensure quality and completeness of data entry 


Step What is it? How can it be used? 
Double-keying With double-keying, you enter the data twice into | You can set up a spreadsheet, such as Excel, for 
a data file, and if the second entry matches the double-keying the data. You can use double- 
first entry, the cell turns green. If the entries do keying with all your data or with a subset of your 
not match, the cell turns red to flag errors for data. 
correction. 


Conditional checks Conditional checks test whether conditions are In Excel, you can use the AND, OR, NOT, and IF 
true or false and make logical comparisons functions to create conditional formulas to verify 
between expressions in data. For example, you if the data entered are accurate. 
might use conditional checks to verify that the 
time a child is reported to have been receiving 
services does not exceed their age. 


Visual inspection With visual inspection, data entry staff compare Visual inspection can reveal mistakes as well as 
the original hard-copy surveys against the entered | areas where additional training may be needed 
data on at least a sample of surveys to check for (for example, if you come across patterns of data 
accuracy and completeness. entry errors). 


Clean your quantitative data 

Data cleaning includes documenting all processes and decisions involved in ensuring that the dataset is complete, 
accurate, and ready for analysis; evaluating and addressing missing data; reviewing data for accuracy; and coding data. 
Document all rules and procedures for handling data issues 


Throughout the process of cleaning the data and creating a complete dataset, it is critical to establish and fully 
document rules and procedures for dealing with any data issues so that you can apply them consistently. This might 
include rules and procedures for 


e How to deal with any issues that you encounter (such as missing data or inconsistent responses) 

e How you will code quantitative and qualitative data for analysis 
Documenting rules and procedures will also help to ensure consistency from year to year. For example, if there are new 
staff, or if current staff do not recall the procedures used previously, there will be a record of the processes. 
Identify and address missing data 


Almost every survey administration will result in at least some missing data, so your rules and procedures should include 
a consistent plan for dealing with missing data. Addressing the problem of missing data is particularly important when 
conducting inferential statistical analyses, but having missing data in a study can potentially affect the results, no matter 
what type of analysis you do. It is not within the scope of this toolkit to go into detail about the different methods to 
treat missing data. However, here are some tips: 


e Decide how you want to handle cases where parents skip large numbers of questions. For example, you may 


want to create a rule that if a parent skips more than half of the items on the survey, their entire survey will be 
coded as missing. 
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e Create and document all decisions and rules for handling missing data and follow them consistently. 


— The easiest way to code missing data is to leave it blank. Using a code like 99 can cause errors in 
calculations.”? 


e Train all data entry staff on the rules so that they know what to do when they encounter missing data. 


The U.S. Department of Education provides guidance on what to do with missing data in experimental studies (see 


https://ies.ed.gov/ncee/pubs/20090049/index.asp). For additional information about what to do with missing data, see 


the resources in Box 3. 


Box 3. Resources for handling missing data 


Howell, D.C. (2012, December 12). Treatment of Missing Data—Part 1. [Web Log Post.]. 
Allison, P.D. (2001). Missing Data. Thousand Oaks, CA: Sage Publications. 


Baraldi, A.N., and Enders, C.K. (2010). An Introduction to Modern Missing Data Analyses. Journal of School 
Psychology, 48(1): 5-37. DOI: 10.1016/j.jsp.2009.10.001 


Verify data accuracy 


Ensuring that your data are accurate is a very important part of preparing for data analysis. Verifying the accuracy of 
your data may require you to eliminate some data (for example, by deleting duplicate or inconsistent data). However, it 
is important that you retain your original dataset, which should include all of the original survey responses prior to 
any changes that you make during data cleaning, to ensure that you have an accurate dataset! Save a new version of 
the dataset (indicating in the file name that it is the clean file) and only eliminate or change data in that file. Ata 
minimum, ensure that you have separate files for the original dataset, the clean dataset, and the final complete dataset 
used for analysis (which might, for example, differ from the clean dataset, if you had to remove a number of duplicate 
entries). 


There are several types of checks you can do to verify the accuracy and quality of survey data.*4 Table 12 outlines some 
questions you can ask about the data, gives ideas for checks you can do to answer the questions, and offers suggestions 
for actions to take to address the issue with the data. As you perform these checks and take action to address any issues 
you might find, remember to only work in a new version of the data file! Keep in mind that many of these checks apply 
primarily to quantitative data, but some of them can also apply to qualitative data. 


23 Carlson and D’Agostino 2015. 
?4 Carlson and D’Agostino 2015. 
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Table 12. Checks you can do to verify data accuracy 


Question about 
(oF) F) 

Do the responses 
indicate that the 
parent should not 
have responded to 
the survey (i.e., 
have the survey 
inclusion criteria 
been violated ?)? 


How might you know the data are accurate? 
Identify the specific criteria that determine 
whether a parent can be included in the data 
collection activities (e.g., does their child have 
an Individualized Education Program [IEP] 
during the school year in question?). 

Use the survey responses or other 
demographic data collected about the parent 
to see whether the inclusion criteria have been 
violated (e.g., no, the child does not have an 
IEP for the school year in question). 

Check for duplicates of the parent’s unique ID 
number. 

Check for duplicate names AND addresses (e.g., 
if there is more than one Jim Smith or Maria 
Rodriguez among survey participants) 

Check that any text you coded numerically 
(e.g., 1 = very strongly disagree) are in 
numerical format in the database file. 

Check that dates are in date format. 

Check that open-ended responses are in text 
format. 

Using your survey as a guide, look to see if the 
way parents responded follows the required 
pattern. For example, if the survey instructs 
parents who answered “no” to question 2 to 
skip question 3 and move to question 4, is 
question 3 actually blank? 


Has a parent 
completed more 
than one survey 
about the same 
child? 

Are all responses in 
the correct 
format? 


Did parents follow 
the skip patterns 
appropriately? 


(Note: This is particularly important for paper 

surveys, but problems with skip patterns can 

also happen with online surveys.) 
e Identify the set of possible responses for each 
question (e.g., ages 3 to 21, a scale of 1-6 from 
very strongly disagree to very strongly agree). 
Review data frequencies (descriptive 
statistics that show counts of parents who 
provided each response for every question) to 
ensure all reported values fall within the set of 
possible responses (e.g., a parent responding 
“50” to a question about his or her child’s age, 
which can only be from 3-21). 


Do any of the 
responses differ 
from the 
predefined set of e 
possible responses 

for each question? 


yiXoh d(oya) 
Remove all of the data for this parent from the 
data file. 


Eliminate the duplicate data from the file. Be sure 
to look at the responses carefully to ensure that 
they are consistent across both entries. If they are 
not, consider following up with the parent to 
obtain the correct responses. 

Correct any format errors in the data file (e.g., 
change “one” to “1’). 


Apply the rules you established about what to do if 
parents don’t follow the skip patterns correctly. 
This might involve deleting the data from question 
3 or taking some other action. 


If possible, follow up with the parent to obtain the 
correct response. If this is not possible, remove the 
response from the data file and count it as missing 
data. 
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Question about 
data 

Are there 
inconsistencies in 
responses among 
questions that ask 
for related or 
similar data? 


Are there outliers 
in the data? 


How might you know the data are accurate? 
Identify any survey questions that ask for 
related or similar data, such as a question that 
asks a parent what grade her child is in and a 
question that asks if her child has already 
created a transition plan after high school. 
Look to see if there are inconsistencies in 
responses to both questions. For example, it 
would be inconsistent if a parent indicated her 
child was in preschool but responded to 
questions about the child having completed a 
transition plan for after high school. 


Review data frequencies and check for values 
that fall outside of the expected range. 
Examine qualitative data to see if there are 
responses that differ greatly from the 
responses of other parents. These responses 
might be an example of an extreme case, but 
they also might provide important information 
about areas for improvement. 


(Note: Outliers are not necessarily incorrect; 
they may reflect data entry errors or an actual 
extreme response reported by a parent.) 


Action 


e For quantitative data, if it is clear that 


inconsistencies are due to an error (e.g., if the 
parent did not follow skip patterns correctly), 
you can eliminate the data for questions that 
should not have been answered. If you cannot 
determine the source of the inconsistency, you 
may have to eliminate all of the inconsistent 
data from that parent’s survey. 

For qualitative data, consider following up with 
the parent to get more information about her 
response. Even if the response is inconsistent, if 
the parent seems to be making an important 
point, consider consulting with stakeholders 
about any implications the response might have 
for policy and practice. 

For quantitative data, check the original data 
source for a possible data entry error. If there 
was no error, apply the rules you established for 
dealing with outliers, which might mean 
removing the response from the data file. 

For qualitative data, consider following up with 
the parent to get more information about his 
response or consulting with stakeholders about 
any implications the response might have for 
policy and practice. 


You may find that data cleaning raises additional questions about specific responses. For this reason, it is important to 
make decisions, or rules, about how you will deal with different types of data issues that arise, document those rules, 
and apply them consistently whenever you encounter issues with the data. 


Code data 


Data coding refers to categorizing survey responses (or responses to other types of data collections) into a defined set of 


values. Your parent survey may include items with different types of response options, such as 


e Yes/No questions 


e Questions answered on a Likert scale (e.g., very strongly disagree, strongly disagree, disagree, agree, strongly 


agree, very strongly agree) 
e Questions with short, specific answers such as the different racial/ethnic groups 


e Questions that result in numerical data (e.g., “How many times did you go to the school last year?” or “How old 
is your child?”) 


You may need to convert these different types of responses to numerical values (e.g., 1, 2, 3, 4, 5, 6) depending on the 
type of analysis you have planned. Many online surveys automatically convert these responses to numerical values, so 
they will be in that format when you download the data. 
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As you start coding your quantitative data, you will need to think about the types or levels of data that you have. 
Numerical data can be divided into four types or levels: 


e Nominal: The data points have no quantitative meaning (e.g., male and female). 


e Ordinal: The order of the data points is meaningful, but the difference between each data point is unknown 
(e.g., data collected with Likert Scales with categories such as very strongly disagree, strongly disagree, disagree, 
agree, strongly agree, very strongly agree). 


e Interval: The order of the data points is meaningful and the difference between each data point is exactly the 
same (e.g., time). 


e Ratio: The order of the data points is meaningful, the difference between each point is exactly the same, and 
there is an absolute zero (e.g., height and weight).7° 


A Note on Coding Survey Data 


While surveys most often generate nominal or ordinal data, the responses to survey questions are often treated as interval data 
and analyzed accordingly. For example, if you use a survey with a Likert scale (such as a 6-point scale from “strongly disagree” to 
“strongly agree”), you can assign numerical values to each category. You can then use these numerical data in your analyses. 
However, if you do this, it is important to keep in mind that the numerical values are arbitrary. You do not really know the 
distance in a parent’s mind between “very strongly disagree” and “disagree.” One parent’s standard for choosing “strongly 
agree” versus “agree” could be very different from another's. If you accept this degree of arbitrariness and assign numbers for 
“very strongly disagree” to “very strongly agree,” as is commonly done, you can produce numerical data using this type of scale. 


When coding quantitative data, make sure you code all data in the same direction. This means that if a question is 
worded such that a high rating means something positive (e.g., very strongly agree), then a high rating should mean 
something positive for every question.*° If questions were originally written so that a high rating (e.g., a score of 6) is 
positive for some items and a high rating is negative for others, you will have to recode some of your data so that it is all 
coded in the same direction (i.e., so that a high rating is always positive). Table 13 provides an example of a survey item 
that was recoded to make it consistent with the coding of other survey items. 


Table 13. Examples of recoding survey items 


Original rating Recoded rating 


This does not need to be recoded because the response 
already has a high rating associated with a positive 
response. 

This item needed to be recoded so that a higher rating 
indicates that the parent feels that the school does 
promote parent involvement. [NOTE: We give this 
example to illustrate why it might be necessary to recode 
data to make it consistent with the other items. However, 
we strongly suggest that you do not use negatively 
worded items with “agree/disagree” response scales 
because they can be difficult for responders to 
understand and interpret.] 


1.1 know my rights as a 
parent of a child with 
special needs. 


6 = Very strongly 


N/A 
agree / 


1=Very 
strongly 
disagree 


2. My child’s school does not 
promote parent 
involvement. 


6 = Very strongly 
agree 


As seen in Table 13, if you want a higher rating to reflect a positive response for all items, you will have to reverse the 
coding for the items where a higher rating actually reflected a negative response. In the example presented in the table, 
a rating of 6 for question 2 actually indicates that the parent does not think that the school promotes parent 
involvement (a negative response), so the coding needs to be reversed so that the response is consistent with the coding 


25 University of California Davis, n.d. 
26 Carlson and D’Agostino 2015. 
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for Question 1 (and a higher rating indicates a positive response). To reverse the coding, you switch the 6 to a 1—or you 
might switch a5toa2ora4toa3. 


Another way to code the data would be to combine responses for some items, such as grouping the responses “strongly 
agree” and “agree” together. In the example of the Likert scale mentioned above, rather than having six possible values 

(ranging from 1-6) you will have three possible values (1 = strongly disagree/disagree, 2 = neither disagree nor agree, 3 = 
agree/strongly agree). 


Prepare your qualitative data 


Similar to the steps discussed in the section on cleaning quantitative data, the first steps in working with your qualitative 
data are entering, cleaning, and preparing the data for analysis. Here are some additional considerations related to 
working with qualitative data. 


e Enter the data into a spreadsheet or qualitative data analysis software program. With some online survey 
programs, you can automatically download your qualitative data along with your quantitative data, which can 
facilitate the process of creating a qualitative dataset. 


e Once you have your qualitative data all in one place, look it over for any data issues, such as missing data, 
responses provided in the wrong field, and misspellings (which will interfere with automated sorting and 
searching). 


e Develop decision rules to provide consistency in the handling of data issues, but we strongly recommend against 
deleting responses! 


e When you identify errors, go back to the original response for clarification or to make a correction. If possible, 
consider going back to the respondent to gain additional clarification. 


A Note on Cleaning Qualitative Data 


When cleaning qualitative data, keep in mind that all responses—especially those that might be considered “outliers” in 
quantitative analysis—potentially can provide valuable insight into the nature and extent of parent involvement in your state. 
“Data issues” specifically refer to problems of missing or erroneous data, not data that might not seem to fit preconceived ideas 
of what the parent involvement data “should” look like. Use caution when applying decision rules. 


Address issues with representativeness after data collection 
Assess representativeness of all collected data 


Once you have completed data collection, you should again assess representativeness of responders—using the steps 
outlined in the section on conducting periodic checks for representativeness—but using the entire dataset. If you have 
been careful when selecting your sample (if you sampled) and have tracked responses, monitored response rates, and 
followed up with nonresponders, there is a good chance that you will have relatively representative data. 


An additional way to assess representativeness is to try to obtain survey responses from a sample of parents who 
initially did not respond to see if the way they respond to the survey differs from the responses you already have. To do 
this, you can follow these steps: 


1. Randomly select a group of approximately 10-20 percent of the parents who did not respond to the survey 
during the initial data collection window 


2. Contact (usually by phone) the parents and ask them for responses to key questions (such as the questions you 
are using to calculate your Indicator B8 percentage) 
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3. Document the responses, depending on the method you used to 
obtain their responses (e.g., either by completing a paper survey for 
them during a phone call or entering their responses into an online 
survey) 


Once you have responses from your group of nonresponders, you can 
compare their responses to the responses of parents who originally 
responded to the survey. In your comparison, examine the degree of 
difference between the two groups to determine if it is meaningful. 


e One option is to use tests of statistical significance between the two 
groups of responders.’ If you decide to use statistical significance as 
your threshold for a meaningful difference, consult with a staff 
member or contractor with statistical expertise to assist with the 
analysis. You can also contact your IDC State Liaison for assistance. 


Remember, there is no one threshold for determining whether data are 
representative. For example, 


e The National Post-School Outcomes Center Response Calculator uses 
a threshold of +3 percent in observed data 


e The ECTA’s Response Rate and Representativeness Calculator uses 
tests of statistically significant differences to determine 
representativeness 


Following up with nonresponders is 
advantageous because it allows you to 
directly assess if there are differences 
between responders and 
nonresponders, but it can cost more 
compared to other techniques because 
you may need to make extra effort to 
get nonresponder parents to respond! 


For example, you may need to use 

e Different survey administration 
techniques, such as using a shorter 
questionnaire or asking parents 
questions over the phone 

e Incentives, such as gift cards or small 
payments, to motivate parents to 
respond 


Keep in mind that some of the 
differences in responses might be due 
to differences in the way that parents 
responded to the survey (the mode). 
You could mention this possible 
limitation of the data when you report 
results. 


If, after all of your efforts, your data are still not representative, you can use techniques such as “weighting” to address 


remaining issues, as discussed in the next section. 


What States Are Doing 


Utah assesses the representativeness of their survey data by comparing the demographic characteristics of the students whose 
parents responded to the survey with the demographic characteristics of all special education students. In 2015-2016, this 


comparison indicated the results were representative: 

e By geographic region where the student attended school 
e By the grade level of the student 

e By the primary disability of the student 


However, parents of White students were slightly more likely to respond than parents of non-White students: 84 percent of the 
parents who returned a survey were parents of a White student, whereas 73 percent of the special education students in the 


sample were White. 


To try to address this issue for the future, Utah held a meeting with a Hispanic consultant at the Utah Parent Center (Utah’s 
Parent Training Information Center) to develop a plan to solicit a higher response rate from Spanish-speaking parents. 


27 Montaquila and Olson 2012. 
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Weight to account for nonrepresentative data 


If you examine your data and find that they are not representative, you may be able to achieve representativeness with 
postsurvey adjustments such as “weighting.” Weighting is a procedure in which data from some groups are assigned 
multipliers so they carry more or less weight to compensate for representativeness shortcomings.” There are several 
types of weighting; here are two that may be appropriate for your purposes:”? 


e Weighting to adjust for unit nonresponse is a type of weighting that is applied to adjust for groups of parents 
who are underrepresented in the dataset. 


— Example: Suppose that the response rate is 80 percent for parents of a child with autism and 75 percent for 
parents of children with other disabilities, which implies that parents of children with other disabilities are 
underrepresented in the dataset. Two weights are then created to adjust the differences in response rates, 
w_i=1/0.8 for parents of a child with autism and w_i=1/0.75 for parents of children with other disabilities. 
The nonresponse adjusted weight is obtained as the product of the base weight and the nonresponse 
adjustment. 


e Poststratification weighting for sampling variance reduction, undercoverage, and unit nonresponse assures 
that sample totals are equal to the external total based on the target population. 


— Example: Suppose, after the nonresponse adjustment, the sum of the weights for males is the same as the 
sum of the weights for females. However, there are more males than females with disabilities in the state— 
65 percent vs. 35 percent. The weights for males need to be inflated (i.e., 0.65/0.5=1.3) by factoring in the 
larger percentage of males with disabilities in the state, and the weights for females need to be deflated 
(i.e., 0.35/0.5=0.7) by factoring in the smaller percentage of females with disabilities in the state, thereby 
restoring the outside population distribution for the weighted sample. 


You can use more than one weighting technique; if you do, the final weight is the product of all weighting techniques. 


We highly recommend consulting a statistician on weighting because it is quite complex. You can also consult your 
IDC State Liaison for assistance. 


What States Are Doing 

Utah uses a standard weighting procedure every year when calculating state-level results to ensure that the parent survey 
results reflect the parent population. Districts, and parents within districts, have an equal probability of selection, so responses 
are weighted by the size of the population of students with disabilities. Thus responses from parents in a district that has four 
times the number of students with disabilities as another district, will receive four times the weight in computing overall state 
results. 


Idaho selects districts for survey participation each year using a 2-year cycle and a stratified process. Stratification is by total 
enrollment, percent Hispanic, geographical region of the state, percent Native American, and special education enrollment. The 
two largest districts are always surveyed in separate years. Results are weighted according to the population size of the district 
to ensure the overall state’s parent involvement percentage is an accurate reflection of the experience of parents of students 
with disabilities ages 3 to 21. 


28 U.S. Department of Education 2012. 
29 DiSogra 2007. 
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Take steps to improve representativeness in future data collection efforts 


If you find that you have had issues with representativeness, it is important that you take steps to improve it in the 
future. For example, if you find that certain groups are underrepresented in your survey responders, in the next year’s 
data collection you may opt to 


e Employ a probability sampling strategy, such as stratified sampling or cluster sampling 
e Include larger subsamples of parents from underrepresented groups (called “oversampling”) 


Keep in mind that if you purposely oversample from certain groups, you will need to weight the data to account for the 
fact that different groups had different probabilities of being selected (also known as having “differential selection 
probabilities”). For example, a state might oversample Hispanic parents to ensure that the sample of Hispanic parents is 
large enough to draw conclusions. However, if the state combines data across ethnicity groups to get estimates for the 
entire population, the state must apply weights to individual values to adjust for the different selection probabilities. 
The state calculates the selection weight for each parent as the inverse of the selection probability. Here, too, we 
recommend consulting a statistician or your IDC State Liaison for assistance. 


To learn more about sampling, see important information about sampling later in this toolkit. 


Now that you have taken steps to clean and code your data and address any issues with representativeness, it is time to 
turn toward analysis. 
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REFLECT: How can we conduct high-quality data analyses? 


As you analyze your parent involvement data, you will definitely need to calculate the Indicator B8 percentage, but you 
should also consider doing other analyses to explore your data and learn more about parent involvement in your state. 
In this section you will learn about 


e Choosing the right method to calculate the Indicator B8 percentage, based on what survey you use 

e Generating descriptive statistics to show patterns in your quantitative data 

e Conducting inferential statistical analyses to test hypotheses about group differences or changes over time 
e Using rigorous qualitative analysis to gain a more nuanced understanding of your data 


e Combining quantitative and qualitative findings to learn more about parent involvement in your state 


Use the right method to calculate the Indicator B8 percentage 


When using survey data to calculate the Indicator B8 percentage for your 
SPP/APR, it is very important to consider the type of survey your state is using. 
The way that you calculate the Indicator B8 percentage will differ depending NGSEANIScale-asit was develonéd, 
on whether your state is using the NCSEAM, a state-developed survey, or a many state surveys are adapted from 
modified NCSEAM. Since all states are currently using surveys, and the majority the NCSEAM scale. Some states have 
of state surveys include Likert-type scales, we focus here on methods you can added their own items and combined 
use with such scales. In this section you will learn how to calculate the these with NCSEAM items. 
percentage using these different types of surveys so that you can be sure that 

you are calculating the percentage correctly and reporting accurate data. 


What States Are Doing 


In addition to states that are using the 


Calculate the B8 percentage using NCSEAM data 


The NCSEAM scale was built on the Rasch measurement framework, and surveys that incorporate the NCSEAM scale 
should be scored using Rasch analysis. Since Rasch analysis requires an understanding of measurement principles and is 
typically conducted using specific software (e.g., WINSTEPS), 7° we recommend working with a measurement expert or 
a third-party contractor if your state is using the NCSEAM or has included some of the NCSEAM items in its state- 
developed survey. Go to the section on Additional Information About the NCSEAM if you want to learn more about the 
NCSEAM scale. 


Here are some key points you need to know to calculate the B8 percentage What States Are Doing 


correctly if you used the NCSEAM:32 ; 
Some state-developed items were 


tested using a Rasch measurement 


e Within the Rasch framework, each parent survey receives a score. 
framework and were found to 


e Then the score on each survey is compared to an established cut-point demonstrate good fit with the 
score. NCSEAM items; these items can be 
used in the Rasch item analyses. 

—  Anationally representative group of stakeholders convened by However, other independently 
NCSEAM recommended using 600 as the cut-point score (they developed items that either do not 
thought 600 was the minimum score to count as fulfilling the intent demonstrate good fit with the 
of Indicator B8). NCSEAM items or have not been 


tested should be analyzed separately. 


30 Linacre 2017. 
31 Data Accountability Center 2008. 
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— States can either adopt the recommended cut-point score of 600 or work with statisticians and stakeholders 
to establish their own cut-point scores. 


e Tocalculate the percentage required for Indicator B8, divide the number of survey responders whose scores are 
at or above the cut-point by the total number of survey responders. Multiply the resulting decimal by 100 to 
obtain the percentage on Indicator B8. 


Calculate the B8 percentage using the NCSEAM Excel Scoring Program 


A group of experts, including two of the developers of the NCSEAM scales, developed an Excel-based scoring program 
(the Schools’ Efforts to Partner with Parents Scale [SEPPS] Excel Scoring Program) that approximates the results achieved 
by doing Rasch analysis, using specialized software programs. The Excel-based program is designed to be used with the 
25-item version of the NCSEAM. Go to the section on Additional Information About the NCSEAM if you want to learn 
more about the NCSEAM scale. The Excel program calculates a score for each completed survey and, applying a cut- 
score of 600, yields a percentage for Indicator B8. Box 4 provides references to the Scoring Program and instructions for 
its use. 


Box 4. References to NCSEAM Survey: Excel Scoring Program 


Data Accountability Center. (2008). Use of the NCSEAM Measurement Scales to Address Reporting Requirements for 


Indicator 8 of the Part B State Performance Plan and Indicator 4 of the Part C State Performance Plan: Frequent! 
Asked Questions. 


Elbaum, B., Celimli, S., and Fisher, W.P., Jr. (2012a). SEPPS Excel Scoring Program. 


Elbaum, B., Celimli, S., and Fisher, W.P., Jr. (2012b). Instructions for Use of the Schools’ Efforts to Partner with Parents 
Scale (SEPPS) Excel Scoring Program. 


It is extremely important that you keep in mind some limitations of the Excel Scoring Program: 


e The Scoring Program is only designed to be used with the 25-item version of the NCSEAM. 


e You can use the Scoring Program if each survey has responses to most of the 25 items. However, if responses to 
many items are missing, the validity of scores produced by the program will decline. 


e If you are using a modified version of the NCSEAM—meaning that (a) you have added some items to the 
NCSEAM scale, or (b) you have added NCSEAM items to your state-developed survey—you cannot score 
responses to the non-NCSEAM items using the Scoring Program. ?2 


A Note on the SEPPS Excel Scoring Program 


The program developers caution that, while comparisons of scores obtained through the Excel Scoring Program and Rasch 
analysis indicate that the Excel Scoring Program produces comparable results, “the extent to which responses from any given 
sample of parents conform to the Rasch measurement model cannot be ascertained through the use of the SEPPS Excel Scoring 
Program,” (Elbaum 2012, p. 2). As such, the authors include the following disclaimer: 


Neither the Office of Special Education Programs, NCSEAM, or the developers of the SEPPS can be held accountable 
either for the results of analyses that utilize the SEPPS Excel Scoring Program or for inferences that might be drawn 
from these results. Consistent with policy related to the SPP, it is the sole responsibility of states to conduct their own 
data quality checks and to ensure the reliability and validity of the results that are reported in the Annual Performance 
Report (Elbaum 2012, p. 2). 


32 Elbaum 2012. 
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Calculate the B8 percentage using a state-developed survey or a modified NCSEAM 


If your state is using its own state-developed survey, or if it has modified the NCSEAM in a way that affects the ability to 
analyze the modified survey using Rasch analysis, there are various ways you might calculate your Indicator B8 
percentage. In this section you will learn two ways to calculate the percentage using basic arithmetic. We include two 
examples based on whether your state is using a single Likert-scale item or multiple Likert-scale items to collect its 
Indicator B8 data. 


A Note on Using Items From State-Developed Surveys to Calculate Indicator B8 


The percentage that you report for Indicator B8 may be more accurate and reliable if you calculate it using the results of 
multiple survey items rather than the results of a single item. Using multiple items allows for a more reliable measurement of 
the true attitudes of parents and protects against response errors or misunderstandings that might occur with using the results 
of only a single question. 


There are many additional advanced analysis methods that you can use to calculate your Indicator B8 percentage, such 
as cluster or factor analysis, analysis of variance (ANOVA), regression, or other modeling techniques such as Rasch 
analysis, which is used for the NCSEAM scale. However, unless you are experienced with these analytic methods, we 
recommend enlisting the help of a statistician or a third-party contractor. 


Calculate a percentage from a single Likert-scale item 


If you are only using a single survey item to report data for Indicator B8, you can calculate the percentage by dividing the 
number of parents who responded to a certain category (or categories, such as “strongly agree” and/or “agree”) by the 
total number of parents who responded to the item, and multiply that number by 100, as illustrated below. 

# of parents who responded to a certain category x 100 


indicator BG Percentage == # of patents who responded to the item 


Table 14 presents the responses to the following hypothetical survey item with six Likert-scale categories and 525 
responders. 


Item 1 [INDICATE YOUR LEVEL OF AGREEMENT WITH THIS STATEMENT]: Over the past year, my child’s 
school facilitated parent involvement as a means of improving services and results for my child 
receiving special education services. 


Table 14. Fictional survey data: Responses to single Likert scale item (n=525) 


Responses (n=525) 
Very Very 
strongly Strongly Strongly strongly 


Item 1 disagree disagree Disagree agree agree 


Percentage of responders 5% 10% 14% 19% 24% 29% 100% 


NOTE: The percentages in all columns do not total 100% due to rounding. 
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To calculate the percentage of parents who chose “very strongly agree” for this hypothetical item, follow these steps: 


# of parents who responded "very strongly agree" 
Indicator B8 Percentage = —————__—__________——— x 100 
# of parents who responded to the item 


150 
Indicator B8 Percentage = 595 * 100 = 28.6% 
Or, if you just want to determine the percentage of parents who responded with some level of agreement to this 
particular item, you may want to add the number of parents who responded “agree,” “strongly agree,” or “very strongly 
agree,” as shown below. 


# of parents who responded "Very strongly agree" + "strongly agree" + "agree" 
# of parents who responded to the item 


Indicator B8 Percentage = x 100 


150 + 125 + 100 
Indicator B8 Percentage = —E 100 = 71.4% 
You can do the same types of calculations for negatively worded items (e.g., “My child’s school does not promote parent 
involvement”). To calculate the Indicator B8 percentage in this case, you would do the calculations using the responses 
that indicate disagreement with the item (i.e., “disagree,” “strongly disagree,” and “very strongly disagree”). 


You can also generate “frequencies” using a statistical software package to quickly obtain the same result. Consult a 
statistician, or your IDC State Liaison for assistance. 


As a reminder, we strongly urge you to use data from more than one survey item to calculate your Indicator B8 
percentage. 


Calculate a percentage from multiple Likert-scale items 


If you are using multiple Likert scale items to collect data for Indicator B8, one method to calculate the percentage 
involves averaging the survey responses for each parent and then comparing the average to a predetermined cut-point 
score that indicates a positiv2e response. You may choose to include in the average all of the items in the survey ora 
subset of items for Indicator B8. Then, report the percentage of surveys (or parents) that meet or exceed the cut-point 
score. To use this procedure, follow these steps: 


1. Assign each Likert category (e.g., “very strongly disagree” to “very strongly agree”) a numeric value (for 
example, 1 to 6) with the numerical values ascending. (See the section on coding data for important 
information about how to correctly code your quantitative survey data). 


2. Seta cut-point score, or the score at which your state feels that a 
school meets the threshold for facilitating parent involvement. 


What States Are Doing 


Some states using a 6-point Likert 


a. Itis important that you select an appropriate cut-point score scale use a cut-point score of 4.0, the 
based on how you have worded your items and defined your numeric value that generally 
categories. corresponds with a rating of “agree.” 


b. You may wish to convene a group of stakeholders and subject- 
matter experts to determine an appropriate cut-point score. 


3. Add the responses across all the items you are using for Indicator B8 to get the total score. 
Total score = Item 1 + Item 2 + Item 3 + Item N 
4. Divide by the total number of items to get the average. 
Total score 


vee = —— — 
Total # of items 
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5. Add the number of surveys that meet or exceed the cut-point score, divide that number by the total number 
of parent surveys, and multiply by 100 to get a percentage that you can report for Indicator B8. 


# of surveys that meet or exceed the cut-point score 


# of parents who responded to the survey oe 


Indicator B8 Percentage = 


Table 15 contains hypothetical survey data from one parent for four items. In this example, the state has set a cut-point 
score of 4.0, which corresponds to the response of “agree” on the 6-point Likert scale. 


Item 1 [INDICATE YOUR LEVEL OF AGREEMENT WITH THIS STATEMENT]: Over the past year, my 
child’s school facilitated parent involvement as a means of improving services and results for 
my child receiving special education services. 


Table 15. Fictional survey data: Parent 1 responses to multiple Likert scale items 


Responses and their assigned values 


Very Very 

strongly Strongly Strongly strongly 

disagree disagree Disagree Agree agree agree 

1 2 3 4 5 

Item 1 v 6 
Item 2 v 4 
Item 3 v 5 
Item 4 v 3 
Total Score 18 
Average 4.5 


Follow these steps to calculate the Indicator B8 percentage for this example: 


1. Assign numerical values to all of the responses to Items 1 through 4 for Parent 1. 


2. Calculate the total score for all four items. 


Total score =6+4+5+3=18 
3. Next, divide the total score by total number of items to get the average. 


18 
Average = ee 4.5 


4. Following this same procedure for every survey you receive (n=500), you determine that 350 out of 500 met or 
exceeded the cut-point score of 4.0. 
5. To calculate the Indicator B8 percentage, add the number of surveys that met or exceeded the cut-point score, 


divide that number by the total number of parent surveys, and multiply by 100. 


50 
500 * 100 = 70% 


Indicator B8 Percentage = 
Now that you have calculated your Indicator B8 percentage, you can move on to thinking about other ways to explore 


your parent involvement data. In the next sections you will see how you might use different analytic techniques to 
explore and get the most out of your data. 
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Generate descriptive statistics to explore your data 


In addition to calculating the Indicator B8 percentage, you can explore your quantitative parent involvement data by 
generating descriptive statistics. Descriptive statistics provide information about the overall trends and distribution of 
the data. This includes reporting percentages or frequency distributions (also called “frequencies”) for nominal and 
ordinal data, as well as reporting summary statistics of interval and ratio data (see the section on coding quantitative 
data for more information about the different types of data). 


Click on the topics below to learn more. 


e Percentages 
e Frequencies 
e Summary statistics 


Percentages. You may want to calculate the percentage of parents who provided responses to questions related to 
parent involvement beyond those questions directly related to Indicator B8. Remember, when sharing the percentages 
(internally or with stakeholders), be sure to also include the total number of people included in the calculation 

(e.g., the number of parents who responded to a certain category and the total number of parents who responded to 
the item). If you sampled, you may consider calculating the percentage of the sample that the responders represent; 
otherwise, calculate the percentage of the total population that the responders represent. Table 16 illustrates one way 
to do this for a fictional parent survey administered to a sample of 799 parents by State A. 


Table 16. Percentage of parents expressing agreement to items on the parent 
involvement survey in State A (Sample = 799) 


Number of Number (and percent) of 
parents who Item responders who agree/ 
responded to response strongly agree/very 
item rate strongly agree 
The teacher communicates with me regularly about my child’s 
764 96% 647 (85%) 
progress 
My relationship with the school staff has a positive effect on my 4 5 
child’s education 741 93% 679 (92%) 
Teachers are available to discuss my questions or concerns 763 95% 594 (78%) 
The school explains what options | have if | disagree with the 5 r 
decision of the IEP team 746 93% 640 (86%) 
| work together with the IEP team as an equal partner to develop . 
my child’s IEP 767 96% 421 (55%) 
IEP meetings are scheduled at times that are convenient to me 784 98% 323 (41%) 


As you examine the data presented in Table 16, there are a number of things that stand out: 
e The item response rate is extremely high, which indicates that State A did a very good job designing its survey 
and making it clear to understand and easy for participants to respond to the questions 


e Even though the item response rates were very high overall, they were lower for certain questions, which may 
indicate a reluctance to respond to those particular questions 


e The percentage of parents who agreed somewhat (i.e., responded agree/strongly agree/very strongly agree) 
differed considerably based on the question. 
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Presenting additional information about the percentage of parents responding to items that go beyond Indicator B8 
gives a fuller picture of the nature and extent of parent involvement in the state, including highlighting areas where 
improvements might be needed. 


Reflect on the Data: Based on the results in Table 16, what changes could State A make to improve parent involvement in 
the state? 


Frequencies. It can also be helpful to look at the frequency distribution of responses to survey items. The distribution is 
the range of values for an item. This provides a complete picture of the variation in responses to the item and can help 
you understand the data better. A simple way to create a distribution is to list every value (or response) of an item and 
the number of parents who provided each response, as shown in Table 17. Analysis software such as SPSS and SAS can 
easily generate tables like this one. You could also include information about the percentage of the total responses that 
each answer represents to give a better idea of where the responses fall. 


Table 17. Frequency distribution of responses to the parent involvement survey in 
State A (Sample = 799) 


# of 
parents 


Very Very who Item 
strongly | Strongly Strongly | strongly | responded | response 
disagree | disagree | Disagree Agree agree to item rate 


The teacher communicates with 


me regularly about my child’s 15 44 58 457 152 38 764 96% 
progress 

My relationship with the school 

staff has a positive effect on my 5 23 34 436 198 45 741 93% 


child’s education 


Teachers are available to discuss 


: 35 60 74 436 150 8 763 95% 
my questions or concerns 


The school explains what options 


| have if | disagree with the 18 32 56 415 120 105 746 93% 
decision of the IEP team 


| work together with the IEP team 


as an equal partner to develop 12 55 279 346 75 0 767 96% 
my child’s IEP 


IEP meetings are scheduled at 


times that are convenient to me 72 155 234 232 91 0 784 98% 


Reflect on the Data: What do the results in Table 17 tell you about how strongly parents responding to the survey feel about 
the efforts their school is making to facilitate parent involvement? 


Summary statistics. Summary statistics are useful for providing additional descriptive information about your data. You 
can calculate summary statistics for interval or ratio data, and for ordinal data (if, as discussed in the section on coding 
quantitative data, you are treating ordinal survey responses as interval data for your quantitative analyses). It is easy to 
calculate summary statistics using software such as Microsoft Excel or statistical software such as SPSS, STATA, R, or 
SAS—as long as you have all of the survey responses entered into your data file correctly (see the section on creating a 
data file for more information). 
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Summary statistics you might consider include 


e Minimum and maximum scores, to show where the responses fall (e.g., from 1-4 or 1-6) 


e Measures of central tendency, which provide an idea of where the majority of scores are located ina 
distribution; these include 


— Mean: the arithmetic average of a set of scores 
— Median: the score above and below which 50 percent of scores fall 


— Mode: the most frequently occurring score in the distribution 


e Measures of variability, which indicate the spread of the scores in a distribution; three measures of variability 


include 


— Range: the distance between the highest and lowest score in a distribution 


— Variance: a measure of how far the scores in the distribution are spread out around the mean 


— Standard deviation: the average distance of scores from the mean (the square root of the variance) 


Table 18 presents these summary statistics for the parent involvement survey administered by State A. 


Table 18. Summary statistics for the State A parent involvement survey 


Std. 
| N | Minimum | Maximum | Mean | Median | Mode | Range | Variance | Deviation 
The teacher communicates 
with me regularly about my 764 Pe 4.05 4 0.880 0.938 
child’s progress 
My relationship with the pols als 
school staff has a positive 
: 741 1 4.26 4 4 5 0.674 0.821 
effect on my child’s 
education 
Teachers are available to 
discuss my questions or 763 eee 3.83 4 Pe] 1.034 1.017 
concerns 
The school explains what 
options | have if | disagree 
with the decision of the IEP 188 aoe . 4 > Lane 1070 
team 
| work together with the IEP 
team as an equal partner to 767 1 5 3.54 4 4 4 0.682 0.826 
develop my child’s IEP 
IEP meetings are scheduled 
at times that are convenient 784 1 5 3.15 3 3 4 1.305 1.143 
to me 


Reflect on the Data: What do the results in Table 18 tell you about the variation in the way parents responded to the 


survey? What do the mean scores tell you? What about the variance and standard deviation? Do their responses tend to group 


together around the mean or are they spread out? 
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For some of the questions about parent involvement your state would like answered (e.g., To what extent do parents 
report satisfaction with their level of involvement in making decisions about their child’s education?), descriptive 
statistics such as percentages, frequencies and summary statistics may be sufficient to get the answers you need. 
However, for more complex questions—such as “Is there a statistically significant difference in the level of parent 
involvement in District A compared to the level of involvement in District B?” —you will likely need to conduct more 
advanced analyses, such as inferential statistical analyses, which we briefly discuss in the next section. 


Test hypotheses with inferential statistical analyses 


As you think about parent involvement in your state, you may want to answer questions that go beyond what you can 
learn from descriptive statistics. Inferential statistical analyses give you more options for learning about your data and 
for testing hypotheses about the nature and level of parent involvement in your state and across districts. 


Table 19 outlines some types of inferential analyses you may want to do with the parent involvement data you collect. 


Table 19. Types of inferential analyses 


Type of analysis Sample questions Keep in mind... 


Test group e ls there a statistically significant difference inthe | e Work with a statistician or an experienced 
differences level of parent involvement in District A quantitative analyst to ensure that you are using 
compared to the level of involvement in the correct statistical test to answer your question. 
District B? 
Do parents in District C and District D report 
equal rates of parent involvement? 


Perform Is there a statistically significant difference inthe | e What groups do you want to examine? Are you 
subgroup level of involvement among African American interested in looking at results by type of disability, 
analyses parents compared to the level of involvement of demographic characteristics of parents or 

Asian parents? students, or school district? 

Is there a difference between the level of e Small cell sizes potentially affect your ability to 

involvement of parents of students with autism report your results. 

compared to parents of students with all other 

disabilities? 
Document Have there been changes in parent involvement If you have been using the same survey with the 
changes over over time in the state overall or in specific same items, you can examine the percentage change 


time districts? overall for Indicator B8 or the percentage of districts 
that have had increases or decreases in parent 
involvement. 

Explain (or Does school staff participation in workshops on These questions involve regression analysis, which is 
predict) improving parent involvement lead to changes in | based on a number of statistical assumptions about 
outcomes the level of involvement reported by parents? your data. Work with a statistician or experienced 
researcher to check if your data meet the 
assumptions. 


It is beyond the scope of this toolkit to go into detail about these and other types of inferential statistical analyses; for 
more information, consult a statistician or an individual skilled in quantitative analysis. If you are interested in learning 
a bit more, Hinkle, Wiersma, and Jurs (2003) and Dimitrov (2010) are good reference books and Rice University, the 
University of Houston Clear Lake, and Tufts University have developed an online statistics book that is free and available 
to the public. You also can find information about statistical analysis online at the Web Center for Social Research 
Methods. 
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Gain a more nuanced understanding using qualitative analysis 


One way to gain a better understanding of parent involvement in your state is to collect qualitative data, which are non- 
numeric data often collected through 


e Open-ended survey items, such as “How does your school encourage you to get involved?,” that ask parents to 
respond in their own words or to provide comments 


e Interviews or focus groups, which are designed to gather in-depth information from parents, teachers, or other 
key stakeholders about the nature and level of parent involvement in a given school or district 


e Observations, which might include visits to school parent nights or professional development workshops 
designed to help teachers facilitate parent involvement 


Qualitative data and qualitative analyses can add richness and depth to your study of parent involvement by, for 
example, allowing you to 


e Understand the level of and factors associated with parent involvement in your state 

e Explore parent perceptions of and reactions to schools’ efforts to facilitate involvement 

e Explore the extent of change in parent involvement as a result of specific programs or practices 
e More thoroughly investigate differences between various strategies to facilitate involvement 

e Establish or confirm facilitators or barriers to achieving greater parent involvement 


e Put findings from quantitative analyses into context, or triangulate quantitative findings 


A Note on Including Qualitative Data in Your Study of Parent Involvement 


It is important to keep in mind that including many open-ended items in your survey, and conducting interviews and focus 
groups or observations, can produce large amounts of qualitative data that you will then need to analyze. And when done 
correctly, qualitative analysis can be quite time-intensive. States interested in including a number of open-ended items in their 
survey, or in collecting qualitative data through another method, should plan for additional resources to enlist the support of a 
skilled qualitative analyst and allow plenty of time to complete analysis before sharing the data or reporting the results. 


Be sure, though, that you will actually DO something with the qualitative data you collect! If you are not willing to invest the 
time and resources into analyzing and using the qualitative responses to your survey, you might as well not include the items in 
the first place. 


In this section you will learn one way to analyze small amounts of qualitative data produced through open-ended survey 
items. If you have collected extensive qualitative data, you should consult with an expert qualitative analyst for guidance 
and support. You can also reference resources available on the OSEP IDEAs That Work website, such as the two-part 


webinar series on planning and conducting high-quality interviews and the Evaluating Special Education Programs: 
Resource Toolkit, both of which offer additional guidance and resources. 


Read the sections on Cleaning and Coding Quantitative Data and Preparing Qualitative Data to learn some important tips 
related to getting your data ready for analysis. Once the data are prepared and in your data file, you can begin your 
analysis. There are many different ways to conduct qualitative analysis depending on which qualitative framework you 
want to use. It is beyond the scope of this toolkit to go into detail about qualitative analysis, but here is one way you can 
analyze qualitative data produced through open-ended survey items: 
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1. Read through all of the data to get a broad, overall sense of the You can conduct your qualitative 
responses and to begin trying to understand them. analysis using spreadsheet software 
such as Microsoft Excel, or you can 


2. After reading through all of the data, go back and methodologically ; ‘ eae : 
invest in qualitative data analysis 


group, or code, responses into similar categories, also known as software such a NVivo, ATLAS.t) 
themes. For example, depending on the questions you have asked, Dedoose, or Ethnograph. There are 
parents may share similar concerns, suggestions, or thoughts about the also free, open-source programs such 
way their schools are trying to facilitate parent involvement. as Coding Analysis Toolkit. Qualitative 
data analysis software partly 
automates coding and helps identify 
themes in the data. Using specialized 
software will allow you to analyze 


Consider the following example: Your survey includes a 6-point Likert- 
scale item that asks parents whether IEP meetings are scheduled at 
times that are convenient to them (response options range from “very 


strongly disagree” to “very strongly agree”). The survey also provides your data in less time than it may take 
space for parents to add written comments related to their answer to to do the same work through 
that question. spreadsheet software, although it still 


may take a significant amount of time 
to set up the program and learn how 
to use it. 


e Then group responses in different categories/themes based on 
similarity, as illustrated in Table 20. 


Table 20. Qualitative analysis example 1a 


Category 1 Category 2 Category 3 
It’s really hard to get to meetings during | would like to have options for meeting No childcare offered at school — what 
the workday. | can’t take time off. times, rather than just being told when it | should | do with my younger daughter 
is! during the meeting? 
| wish they wouldn’t hold meetings Teachers don’t give us enough time to Aftercare doesn’t cover meeting times, 
during rush hour — it takes me an hour plan to attend — one week’s notice isn’t so if | can’t bring my other child with me, 
just to get there! enough. | can’t go to the meetings. 
The bus schedule doesn’t allow me to get 
to the meetings on time. 
Buses don’t run that route at meeting 
times. 


e Or you might break the categories down even more, as shown in Table 21. 


Table 21. Qualitative analysis example 1b 


Category 1 Category 2 Category 3 Category 4 


It’s really hard to get to Bus schedule doesn’t allow | would like to have options No childcare offered at school 

meetings during the workday. | me to get to the meetings on for meeting times, rather than | — what should | do with my 

| can’t take time off. time. just being told when it is! younger daughter during the 
meeting? 

| wish they wouldn’t hold Buses don’t run that route at Teachers don’t give us enough | Aftercare doesn’t cover 

meetings during rush hour —it | meeting times time to plan to attend — one meeting times, so if | can’t 

takes me an hour just to get week’s notice isn’t enough. bring my other child with me, 

there! | can’t go to the meetings. 
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3. Create names or labels for categories/themes. As you group, you will usually start to see common themes 
emerge (assuming you have enough responses), so you can create names or labels to describe or explain what 
the data are telling you. 


In the example presented in Table 20, the labels for the themes might be 


e Category 1: difficulty getting to meetings 
e Category 2: issues with not being consulted about the date or time of the meetings 
e Category 3: lack of childcare options for siblings during meeting times 


In the example presented in Table 21, the labels for the themes might be 


e Category 1: difficulty getting to meetings during the workday/rush hour 

e Category 2: issues obtaining transportation to the meetings 

e Category 3: issues with not being consulted about the date or time of the meetings 
e Category 4: lack of childcare options for siblings during meeting times 


4. Identify associations and relationships within the themes or across open-ended questions. This step will 
depend on the amount of qualitative data that you have and the variety of responses that you receive. 


Returning to our example, suppose that your survey also asks parents to provide additional comments related to the 
item “Teachers are available to discuss my questions or concerns.” You obtain the responses presented in Table 22. 


Table 22. Qualitative analysis example 2 


Category 1 (@r-} -1:{0) AY ay 


| really appreciate how much teachers work to make Teachers don’t give us any choice in the IEP meeting dates or 
themselves available. times, and | can’t take off work during the day, so | don’t have 
many opportunities to talk with my son’s teacher. 


It is great that my son’s school gives us options in the My daughter’s teacher only holds IEP meetings during the day 
scheduling of IEP meetings. They give us three dates andtimes | or right after school, so | can never get there. If he could hold 
and allow us to choose one. That makes it really easy for me to | meetings in the evening, it would be a lot easier to talk with 
go to the meetings and have time to talk with his teacher. him about her issues. 


In Table 22, you can see that the comments included in category 1 contrast with the comments presented in category 2. 
In addition, the responses here raise similar issues to those that were raised in the responses to the question about the 
scheduling of IEP meetings (presented in Table 20 and Table 21). 


With the help of a good qualitative analyst, you can explore your qualitative data further to identify other such 
relationships and linkages to gain a better understanding of the nuances of your data. 


5. Take a step back to consider what the analyzed data mean and to assess how they relate to the evaluation 
questions under study. You might also ask if the patterns in your qualitative data are similar to or different from 
your quantitative findings. 


6. Return to the data, groupings, and labels as often as necessary until you feel confident that the groupings 
accurately reflect the major patterns of the comments and that your coding scheme captures most of the 
comments. Even when you feel your coding process is complete, review the data once more for patterns, 
trends, and common themes that emerge. 
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Box 5 presents some resources for analyzing qualitative data that you might find useful. 


A Note on Analyzing Open-Ended Survey Items 

Research suggests that open-ended items will generally take more effort for parents to complete, so parents with strong 
Opinions about an item are most likely to complete these open-ended items (Leeuw, Hox, and Dillman 2008?°). This is 
important because your qualitative data may not actually reflect the experience of all parents, but are more likely to 
reflect the views of parents who have higher levels of either positive or negative attitudes. 


Box 5. Selected resources for analyzing qualitative data 


Berkowitz, S. (1997). Analyzing Qualitative Data. In J. Frechtling & L. Sharpe (Eds. 
Mixed Method Evaluations. Arlington, VA: National Science Foundation. 


Maxwell, J.A. (2005). Qualitative Research Design: An Interactive Approach (2nd ed.). Applied Social Research 
Methods Series, Vol. 41. Thousand Oaks, CA: Sage. 


Suter, W.N. (2012). Qualitative Data, Analysis, and Design. In Introduction to Educational Research: A Critical 
Thinking Approach (2nd ed. . 342-386). Thousand Oaks, CA: Sage. 


Combine quantitative and qualitative findings to enhance learning 


One of the benefits of including both quantitative and qualitative methods in your study of parent involvement is that 
you can get a more in-depth understanding of the nature and extent of parent involvement in your state. 


e Quantitative data can quickly show the level of parent involvement in your state and, if your data collection and 
analysis activities are designed for this purpose, can even show patterns of differences across districts, schools, 
demographic characteristics, etc. 


e Qualitative data provide richness and depth that can enhance your understanding of the efforts to encourage 
parent involvement across your state and identify facilitators and barriers to such involvement. 


When you present your quantitative findings (e.g., tables/graphs with percentages or frequencies), for instance, you 
could supplement those findings with any findings that emerged from your qualitative analysis of open-ended survey 
items to provide additional information and add context to the findings. 


For example, Table 16 presented the results of a fictional parent involvement survey in State A. In that table, one item in 
particular stood out from the rest because, while most parents agreed with the other items, the majority of parents did 
not agree with that item. Specifically, 59 percent of parents did not agree that IEP meetings were scheduled at times 
that are convenient to them. But, since State A asked parents to provide comments related to their response to the item 
and several parents did so (as illustrated in Table 20 and Table 21), State A has additional qualitative data that can help 
to explain the quantitative results. Qualitative analysis identified four themes associated with why parents do not feel 
IEP meetings are scheduled at convenient times: 


e Difficulty getting to meetings during the workday/rush hour 
e Issues obtaining transportation to the meetings 


e Issues with not being consulted about the date or time of the meetings 


e Lack of childcare options for siblings during meeting times 


33 Leeuw et al. 2008. 
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By gathering both quantitative and qualitative data, State A now has a better understanding of why parents do not think 
that IEP meetings are scheduled at convenient times, as well as information about additional barriers affecting parents’ 
ability to attend the meetings. This understanding may encourage State A to talk with local districts and schools to see if 
they can find ways to make it easier for parents to attend. 


Now that you have learned about ways you can conduct high-quality analyses to gain a better understanding of your 
state’s data, you are ready to think about what the data mean and how you can use it to improve results for students. 
Then you can share the results with key stakeholders. The next section, REACH, will show you how you can make 


meaning of the findings and effectively communicate with different audiences to encourage improvements in programs 
and processes. 
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REACH: How can we make the most of our parent 
involvement data? 


There are many reasons for your state to collect and analyze parent involvement data that go beyond the federally 
mandated requirement to report on Indicator B8. Table 23 presents some ways that states can use parent involvement 
data to ensure that they are meeting the needs of students with disabilities and their families. 


Table 23. Ways states can use parent involvement data 


Possible use | Reason 

Ensure high-quality Communicating parent involvement data can help to ensure that the services provided to encourage 

services are provided parent involvement are high quality, relevant, and useful. States can use the data to monitor and 
manage implementation of parent involvement initiatives and provide feedback to inform 
improvements, with the ultimate goal of improving services and results for students with disabilities. 

Promote Keeping key stakeholders (such as OSEP, district and school leaders, advocacy groups, school staff, and 

accountability parents) aware of how well states and districts are doing in their efforts to encourage parent 
involvement in the education of students with disabilities can serve as a mechanism for accountability. 
When results show that parent involvement is low, such information can serve as a stimulus to 


improve. 
Provide important Communications about successes and lessons learned through parent involvement data can be 
information about valuable to others doing similar work and to a wider audience of individuals interested in learning 
what works (and about the results of investments to increase parent involvement. 


doesn’t work) 

Generate support for Communications about how involved parents feel can help generate interest in, and support for, 

services projects to support parent involvement. Conversely, the impact or reach of efforts to encourage parent 
involvement can be limited if states and districts fail to effectively communicate about achievements 
and results of their investments. 


In this section you will learn how your state can 


e Report the correct information related to Indicator B8 to OSEP 
e Go beyond reporting only on Indicator B8 to make meaning of the broader parent involvement data you collect 
e Use results of data analyses to spur improvements to programs and services for students with disabilities and 


their families and inform action 


How Can Stakeholders Help? 
Ask stakeholders to review the results of your data analyses and participate in a larger decisionmaking process around overall 
program improvement and priorities. 


Report the correct information to OSEP and the public 


OSEP is one of the primary audiences for the results of states’ Indicator B8 data collection and analysis activities. As part 
of their State Performance Plan/Annual Performance Report (SPP/APR) submission to OSEP, states are required to 


e Set annual targets for Indicator B8, as defined with stakeholder input, and report on the state’s progress toward 
meeting those targets 


e Report on the APR results for Indicator B8 from the previous years 
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e Describe any changes to the Indicator B8 data collection process, including the survey (or other method if 
applicable) being used to collect data 


— If using a new survey, states must submit the new survey along with the APR 
e Explain how the state ensured that data collected are valid and reliable 


e Describe how well parents who participated in data collection represent all parents of students with disabilities 
in the state (see the section on ensuring data are representative for more information) 


— If certain groups of parents were less likely to respond to the survey than other parents were, states should 
include which groups were under- and overrepresented and how they addressed issues with 


representativeness, such as weighting the data or implementing strategies for increasing response rates. 


States are also required under IDEA to report annually to the public on the performance of each local education agency 
(LEA) located in the state on the targets in the state’s performance plan (Section 616(b)(2)(C)(ii)(I)). When LEAs are not 
represented in the survey (or other type of data collection) for a particular year because the state draws a sample of 
LEAs each year or because no parents in the LEA responded, states should report the most recently available result for 
the LEA and the date the data were obtained.*4 


In addition to posting the APR on the state website for public access, you should consider posting additional information 
regarding Indicator B8 directly on the state website, such as response rates, comparison of results to state and LEA 
targets, and any identified trends. 


Go beyond reporting on Indicator B8 to make meaning of the data 


While it is essential to report on Indicator B8, states should also consider taking 
a deeper look at the parent involvement data they collect and examining it to 
make meaning. There are a number of ways you might do this, including 


What States Are Doing 


Several states have created resource 
guides or similar documents to assist 
LEAs in understanding the data and 
improving results. Some states’ 

e Looking for patterns in the data resource guides include strategies for 
improving results for every question in 
that state’s survey, such as links to 


e _ Linking back to questions you want answered 


e Examining or comparing performance statewide 


e Exploring hypotheses articles or briefs with strategies for 
encouraging parent participation in 
e Drawing conclusions from the data IEP meetings or templates related to 


survey items about parent input and 
participation that can be used for 
recording parent input during an IEP 


Link back to questions you wanted to answer. When examining your data, it is 
usually good practice to start by returning to the questions that you were trying 


to answer in the first place. Think about what your state was trying to learn meeting. States have also developed 
through its data collection activities and look at the data through that lens. lists of probing questions for LEAs to 
use when reviewing their data to 
e Did the state focus solely on gathering data related to Indicator B8? focus on the strengths, challenges, 
and reasons results may vary by 
— Is the state meeting its targets related to parent involvement? group. 


— Are some groups of parents more involved than others? 


e Did the state gather other types of data that can be used to inform state and local activities to encourage parent 
involvement? 


— What do those data say about the nature and extent of parent involvement in the state? 


— Aresome types of activities more effective than others? 


34 U.S. Department of Education 2017. 
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— What factors are barriers or facilitators of parent involvement? 


e If your state involved other individuals, besides parents, in its data collection (e.g., teachers, school staff), what 
did you learn from the other participants? 


— How well do the perceptions of parents about their level of involvement align with the perceptions of the 
other participants? 


Look for patterns. As you work to answer your initial questions, you may also wish to look for patterns in the data that 
you may (or may not) have anticipated. One way to do this is to graph your quantitative data or arrange your qualitative 
data in a matrix, as sometimes visual representations of the data can reveal features that are otherwise difficult to 
identify.*° Consider the example presented in Figure 9, which shows a graph of the responses to two items on State A’s 
parent involvement survey in District A and District B. 


Figure 9. Graphs of parent responses to two questions on State A’s parent 


involvement survey 
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35 National Forum on Education Statistics 2016. 
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In Figure 9 you can see that parents in Districts A and B responded in basically the same way to the item “The teacher 
communicates with me regularly about my child’s progress.” That is, most parents agreed or strongly agreed with the 
item. However, you can also see that the responses to the item “IEP meetings are scheduled at times that are 
convenient to me” differed considerably. Specifically, parents in District A overwhelmingly disagreed with the item, 
while parents in District B had mixed responses to the item, with some disagreeing and some agreeing. 


Now look at the example of a qualitative data matrix presented in Table 24. 


Table 24. Data matrix comparing qualitative responses to State A’s parent 
involvement survey in three districts 


Participants’ views of activities to encourage parent involvement in three districts 


Respondent 
group 
District A 


Activities named Which are most effective? Why? 


Provided practical tips and 
helped develop skills 

e Concise way of communicating 
information 


Workshops 
Presentations 


Newsletters 
e Community Events 
e Workshops 

Presentations 


District B Workshops Workshops Provided practical tips and 
Parent Information Packets Parent Information Packets helped develop skills 
Newsletters Provided individualized 

information 

District C Workshops Workshops Provided practical tips and 
Referrals e Referrals helped develop skills 
Resource Fairs e Increased awareness of 
Newsletters services 
Community Events 

SOURCE: Adapted from Berkowitz 1997. 


As seen in Table 24, the data show that all parents consistently rated workshops as the most effective activity districts 
offered to encourage parent involvement. In this example, however, the reasons parents gave for why they felt the 
workshops were the most effective differed across districts. Arranging the data in this way can allow for the emergence 
of patterns in the data that you might not have found if you had simply repeated the data in narrative form. 


Reflect on the Data: What are the advantages/disadvantages of presenting quantitative and qualitative findings such as 
those presented in Figure 9and Table 24? What different types of information can you learn? 


As you examine your other data related to parent involvement across districts or schools, you may see patterns emerge 
that provide insight into areas of success or areas where improvements may be needed. 


Examine or compare performance. Another way to use your parent involvement data is to examine or compare 
performance relative to certain standards or across groups. This might involve looking at parent involvement data across 
districts and schools to see if your state is achieving targets overall, or investigating how one district compares to 
another. Sometimes you might think you see clear differences in performance just by looking at graphs of the data, but 
in many instances it is a good idea to perform inferential statistical analyses to see whether those differences are 
actually statistically significant. For example, look at Figure 10 and Figure 11. Statistical analyses confirm that there is no 
difference in performance between Districts A and B in Figure 10, and they also show that there is a difference between 
their performances in Figure 11. 


www.ideadata.org 58 


IDC Parent Involvement Toolkit 


Figure 10. Comparing parents’ responses in Districts A and B 
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Figure 11. Comparing parents’ responses in Districts A and B 
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Of course, using statistical analyses when a difference in performance is not clear from just looking at a graph is not the 
only reason to do such analyses. Other reasons include 


e Providing stronger evidence of an actual difference in performance to support requests for new or continued 
funding 

e Demonstrating the efficacy of particular strategies or activities to improve parent involvement 

e Getting a better understanding of the magnitude of differences in performance 


Explore hypotheses. Similarly, you can explore hypotheses with your data, such as looking at whether one group of 
parents is more involved than others, or whether certain types of activities are associated with higher levels of parent 
involvement. See the section on exploring hypotheses with inferential statistics for more information about using your 
quantitative data in this way. 
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Some researchers advocate using qualitative methods to explore hypotheses 


and investigate causal processes (e.g., Donmoyer 2012a, 2012b; Maxwell How Can Stakeholders 


2004a, 2004b, 2011, 2012). Specifically, they say that qualitative researchers Help? 
can draw and support causal conclusions by focusing on the causal processes Sharing results with stakeholders can 
that result in particular outcomes. You may want to consider integrating case help you to draw conclusions. 


Stakeholders might quickly identify 
patterns or understand the meaning 
behind certain findings. In this way, 
making meaning from your data can 
be an iterative process where you 


studies to investigate notable examples of success (or failure) encouraging 
parent involvement that come out of your data analyses to see what lessons 
you can learn and apply to different settings. 


Draw conclusions. After you have taken a close look at your data in order to explore different hypotheses through 
answer your original questions, identify patterns, examine performance, and graphing or conversations with 
explore hypotheses, you can draw conclusions about what it all means. When stakeholders until you feel confident 
drawing conclusions about your data, it is important to keep in mind the quality that you understand the true meaning 


of the data and any limitations associated ‘with the data, such as whether or of your data. 


not they are representative.*° Think about the implications of what you learned 
for policy and practice and consider how you will share the data with key stakeholders to spur improvements (if needed) 
and inform action. 


Use results to spur improvements and inform action 


Arguably, the most important reason to collect parent involvement data is to gather information that can lead to 
improvements in the delivery of services and supports to students with disabilities and their families. But collecting (and 
analyzing) data is not enough—you must apply (and share) what you learn if you want to make the most of your efforts 
and, ultimately, effect change. Here are some specific ways that you might use your parent involvement results: 


e Toinform monitoring and the annual determination process 
e To identify and prioritize needs for strategic and improvement planning 
e Tocreate and support family engagement initiatives 


Inform monitoring and the annual determination process. States can use P 

parent involvement data to inform special education monitoring priorities, and What States Are Doing 

they may include the data as part of the annual determination process for each pace icrearealeninimecnecione cone|p 

district, with a focus toward improving outcomes for students with disabilities. districts as they conduct a review or 

As part of the monitoring or determination process, consider self-assessment of their parent 
involvement data. 


In the context of monitoring, states 


e Reviewing parent involvement data disaggregated by student 
characteristics, such as age, gender, disability, and race/ethnicity, to see if parents of some groups of students 
are reporting higher or lower levels of engagement 


e Examining parent involvement data by district (or by region in states with small districts) to identify specific 
issues or areas of need 


This information can help to identify possible ways to target professional development or create activities targeted at 
improving involvement among certain groups of parents and students. 


36 National Forum on Education Statistics 2016. 
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Identify and prioritize needs for strategic and improvement planning. States 
can also use parent involvement results to identify and prioritize needs as part 
of their strategic planning and improvement processes. For example, you can 
use your parent involvement results to 


What States Are Doing 
Many states reviewed their parent 
involvement results when they 
developed their State Systemic 


: _ : Improvement Plan (SSIP) and used the 
e Identify areas throughout the state where additional technical deca cain orn dies ca eae andl 


assistance (TA) or professional development (PD) may be needed—and evidence-based practices they 
develop a plan to support districts as they implement the planned to implement. 
recommendations 


e Inform the choice of strategies or evidence-based practices that schools can use to improve their relationships 
with families 


Create and support family engagement initiatives. You can also use the parent involvement data to support any 
ongoing family engagement initiatives in your state or to create new initiatives. Here are some ways you might use 
parent involvement findings to inform and support your state’s overall family engagement activities: 


e Apply what you learn to develop family engagement teams and frameworks with districts and at the regional 
level 


e Include family engagement as a priority for any discretionary grants your state sponsors so that districts have a 
vehicle for funding PD to improve family engagement 


e Inform development and/or use of curriculum modules on building family and school partnerships 


e Consider examining the Indicator B8 parent involvement data in conjunction with the data collected from 
parents to measure family engagement as required by Title |,?” and use the results to report on the strengths 
and areas in need of improvement more broadly 


What States Are Doing 


Georgia’s statewide Parent Mentor Partnership program (http://www.parentmentors.org/), funded under the Georgia Department 
of Education’s IDEA grant and local funds, is a network of over 100 parent mentors who partner with special education directors 
in over 90 school districts to embed family engagement into school and district initiatives. The program uses the results of the 
parent involvement survey to develop annual priorities specific to the parent mentor program and to local programs as well. 
Participating districts and the districts’ parent mentors receive a detailed analysis of their district’s parent involvement data. 
Parent mentors collaborate with district special education directors to develop, based on the results of the survey, goals and 
initiatives for enhancing family engagement and outreach activities toward improving outcomes for parents and families of 
students with disabilities. The state provides guidance to all parent mentors regarding the use of the results of the survey, and 
parent mentors collaborate with state staff to share ideas and strategies to increase positive results. 


Pennsylvania’s verification process for preschool special education programs, by the Bureau of Early Intervention Services, Office 
of Child Development and Early Learning, includes not only compliance indicators but quality indicators which lead to a 
continuous improvement process. As part of this work, the state has strengthened their focus on family engagement. Specifically, 
local programs are required to review their family survey data and, if there are concerns, address them in their Quality 
Enhancement Plan (QEP), which focuses on specific programmatic changes or outcomes and includes information on how change 
will be measured. State early intervention advisors and Early Intervention Technical Assistance staff provide technical assistance 
and support for this process. 


Box 6 presents some additional resources that might help your state to make improvements based on its parent 
involvement data. 


37To meet the parent involvement provisions of Title |, states, districts, and schools are required to implement family engagement activities and frequently collect 
data from parents to measure family engagement. 
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Box 6. Additional resources to help states use parent involvement data to make 
improvements 


Wisconsin’s Indicator 8 Family Survey — School-Aged Strategies and Resources for IEP Teams to Engage Families 
contains strategies and resources to assist IEP teams with engaging families to improve results for Indicator B8. 
https://drive.google.com/file/d/OB7tictHsg-E8bXMwNmRZQIZ6T2c/view 


The Pennsylvania Training and Technical Assistance Network (PaTTAN), an initiative of Pennsylvania Department of 
Education’s Bureau of Special Education (BSE), works in partnership with families and LEAs to support programs and 
services to improve student learning and achievement. PaTTAN’s website offers a number of resources for parents and 
LEAs related to using Indicator B8 data. 
http://www.pattan.net/category/Educational%20Initiatives/Family%20Engagement 


The National Center for Special Education Accountability and Monitoring (NCSEAM) developed seven modules based 
on evidence-based practices to help schools improve performance on Indicator B8. The modules correspond to items 
on the NCSEAM survey and include General Communication Strategies, Parent Friendly IEP Practices, Home School 
Note, Behavior Strategies, Reading Strategies, Mathematics Strategies, and Homework Strategies. 
http://cyfs.unl.edu/futures/ncseam_guidelines.html 


These are just a few of the ways you might use the data you collect to inform your state’s efforts to facilitate parent 
involvement. Remember, stakeholders can serve as important resources during this time. Your state’s Parent Training 
and Information Center (PTI) and Community Parent Resource Center (CPRC) can be particularly helpful in identifying 
ways to apply what you learned from your analyses to improve or develop new strategies to increase parent 
involvement throughout your state. 


In the next section you will learn some strategies you can use to increase the likelihood that the results of your efforts 
reach your intended audiences and can be used by them. 
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REACH: What is the best way to communicate findings to 


key audiences? 


To make the most of your parent involvement data collection and analysis 
activities, you must effectively communicate your findings. There are many 
ways to communicate your results, such as 


e Communicating internally among staff who are responsible for (or 
overseeing) data collection, analysis, and reporting to help them 
improve activities going forward 


e Reporting your Indicator B8 percentage to OSEP 


e Highlighting successes or challenges for local district or school staff, or 
for staff at your state’s PTl and CPRC 


e Telling parents about how being more involved can help improve their 
child’s educational outcomes 


It is beyond the scope of this toolkit to go into detail about communicating 
findings, so here we will briefly talk about two steps that you can take: 


e Develop a communication plan 
e Tailor your communications to meet audience needs 


For more information, tools, and strategies to help you effectively 


communicate your findings, check out Effectively Communicating Evaluation 
Findings, which was developed by the Center to Improve Project Performance 


5 Tips for Effectively 
Communicating Parent 
Involvement Findings 


al 


Consider what your audiences 
most want to know about parent 
involvement, in addition to the 
data and information you most 
want to share. 

Create your data collection/ 
analysis and communication plans 
early so that each can inform the 
other. 

Be succinct. 

Plan to use multiple modes or 
techniques for communication. 
Rarely is one technique or tool 
enough to fully reach your 
audiences. 

Investigate cost-effective means 
of communicating such as social 
media and web-based platforms. 


(CIPP) and is available on the OSEP IDEAs That Work website along with a number of other resources related to 


evaluation. 


Develop a communication plan 


A communication plan can help you to think through 


e What you need (or want) to share 


e Who to share it with 


e Which formats you will use to communicate information to your different audiences 


e How frequently you will share it 


One of the first steps in developing an effective communication plan is to identify the audiences. Audiences can include, 


among others, 


e OSEP 


e State or other (e.g., third-party contractor) staff who are working on the parent involvement data collection and 


analysis activities 


e State staff providing services to students with disabilities and their families 


e __ District and school staff 


e Parents who are participating in the data collection activities or who might be asked to participate in the future 
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e Individuals or organizations that support and promote parent involvement activities and initiatives (e.g., parent 
advisory groups, PTls, CPRCs) 


e Parents and families who need or want to learn more about how to increase their involvement in their children’s 
education 


e State advisory panels or state boards of education 


As you plan to communicate your findings, it’s useful to answer three questions for each audience: 


1. What does the audience need (or want) to know? 
2. How will the audience use the information? 
3. What is the best way to communicate information (in terms of timing and format)? 


Once you have obtained information about your audiences, their needs, and how they will use the data, you can start to 
develop a comprehensive communication plan. You can create one overarching plan or a plan tailored to each audience. 
Here are some factors to consider when creating your plan: 

e What types of data or information are most compelling to each audience? 

e How much data are needed for each audience? 

e What is the best time to communicate your findings to different audiences? 


e What specific method of communication (e.g., infographic, webinar or other presentation) is best for clearly and 
accurately communicating with different audiences? 


e What are the cost considerations associated with collecting data to address the specific needs of different 
audiences and using different methods to communicate with these audiences? 


e What is the best way to ensure that your target audience will receive and understand your intended message? 


Ideally, you should conduct this process of going through and identifying potential audiences and their needs for data 


related to parent involvement when you are planning your parent involvement data collection activities. If you start 
early, you can ensure that your parent involvement data will answer questions that are meaningful to the identified 


audiences and contain the types of data they find convincing. 
Tailor communications to audience needs 
As you learned in the section on developing a communication plan, it’s important to think about audience needs and 
tailor your communications accordingly. Here are some questions to help you think through how to do this: 
e What might a particular audience need or want to know about parent involvement in your state? 
e How will they likely use the information? 
e What data presentation format(s) might they find convincing? 
e What tools are typically used to communicate with them? 
e What is the typical frequency of communications? 


Using the questions listed above as a guide, Table 25 gives an example of a strategy sheet you can create to tailor your 
communications to a particular audience, in this case district or school staff. As you think about your audience, you can 
check the boxes that correspond to the needs and preferences of that particular audience. 
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Table 25. Strategy sheet for tailoring communications to target audiences 


What might they 


need or want to How will they likely Compelling data Method of Frequency 
Target know? use the information? presentation formats communication '(@at-vel.ar-) 
audience (Check all that apply) (OY =Yol a=) | a ar-yer-] °) °) \V2) (Ol aX=Yolar-) | a at-yar] 9) °) VA) (Check all that apply) ivefe}are] 9) ) iA) 
District or Levels of parent Maintain efforts to Tables or graphs of Infographic Weekly 
school staff involvement encourage parent quantitative data Executive summary/ Monthly 
o statewide involvement Visual tools such as synopsis Semi- 
o by district Increase efforts to GIS maps showing Presentation or annually 
o by school encourage parent involvement by webinar Annually 
o by subgroup (e.g., involvement geographic area Journal article Other: 
race/ethnicity, Suggest changes to Qualitative analysis of Interim/final report 
disability or use new parent parent perception Technical report 
category) involvement data Other: 
Parent perceptions strategies Staff/parent vignettes 
of their level of Other: Other: 


involvement, 
facilitators/ 
barriers to 
involvement 
Parent reactions to 
district/local efforts 
to increase 
involvement 

Other: 


You can create a similar strategy sheet for each of your audiences. When thinking about how to tailor your 
communications, consider what data you have (or will have) available after data 

collection is complete. You might consider including the data source(s) in your What States Are Doing 
communication plan to show which data will be used to communicate which 


ay : : States have used different approaches 
findings or to which audience(s). 


to report on the progress of each LEA 
toward meeting the state targets in 
their SPP/APR (Section 
616(b)(2)(C)(ii)(1)). 


Here are some things to consider when sharing your findings: 


e For each audience, think through exactly what data are the most useful 
and meaningful to that group, as well as what data will elicit insight 


from the group, leading to a greater understanding of the results. eo Most states post EEN alndaliiepors 


on the state’s public website. 
e Select the key elements that you want to communicate, such as a few e Some states create a “report card” 
critical items that convey a key message or tell a relevant story, and or dashboard for each LEA that 


focus primarily on them. displays results for each APR 
indicator, including Indicator B8. 


e Present the data in a visually appealing format—following data Report cards typically contain 
visualization principles and techniques to help each group understand information about the state target 
the results without overwhelming them. and the current year of data for the 


eee , . state and the LEA. Sometimes the 
e Present any limitations or data quality issues. For example, if the report cards also have the previous 


response rate is low, note that the results only represent the opinions of year of data to present any 
some portion of all parents of children with disabilities. progress or slippage. 


eo : ‘ : : eer e Other formats for LEA reports 
Additionally, here are some important considerations when sharing findings AEINdEIPDE yenaltsarganized By, 


with two key stakeholder groups: parents/families and educators. indicator and then by LEA. 


www.ideadata.org 65 


IDC Parent Involvement Toolkit 


Parents/families 


It is good to provide results of the parent involvement survey as part of presentations or resources available 
during planned meetings, regional trainings, and other events for parents of students with disabilities hosted by 
the state, districts, or Parent Centers. 


— For example, districts or Parent Centers may include results in newsletters that are posted on the 
districts’ or Parent Centers’ websites, distribute the results at events and activities via presentations or 
brief summaries, or email the survey results to families who are part of a listserv. 


Communications should focus on results most relevant to parents, such as parents’ level of satisfaction with 
their children’s progress and services, involvement in the process, any changes in parents’ attitudes over time, 
and plans for addressing areas in need of improvement. 


Communications should be accessible, both in terms of Section 508 compliance and in the language used in the 
communications. For example, if sharing results with parents who do not speak English, develop materials in 
their primary language and have interpreters available during meetings. Use “parent-friendly” language, 
avoiding jargon and acronyms, and keeping in mind the average reader is at a sixth- to eighth-grade reading 
level.38 


Educators, such as teachers, related service providers, administrators, and family engagement specialists 


Include results of the parent involvement data collection activities as part of regular communications with 
educators, such as newsletters or emails. 


Share results with district committees including the Parent Teacher Association, data teams, or parent 
engagement groups. 


— Consider the group’s needs in the presentation of results. Some groups, such as data teams, may 
benefit from obtaining the complete results, while other groups may benefit more from seeing results 
related to high-level areas of need that are the focus of the state’s or district’s improvement efforts. 


Consider developing internal (nonpublic) local dashboards for each district or school to share the district or 
school results for Indicator B8 as well as the data collection instruments used. 


— Consider presenting the results disaggregated by demographic characteristics (e.g., child’s disability, 
gender, and race/ethnicity) and include a comparison of the district/school results with the 
district/state-level findings, as well as the number and percentage of surveys received. 


Have districts with improved or high-quality parent involvement data share successful strategies at statewide 
meetings of districts. 


Box 7 presents additional resources you can use to learn more about effectively communicating your findings with 
stakeholders. 


38 DuBay 2004. 
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Box 7. Additional resources about communicating findings 


The Center to Improve Program and Project Performance (CIPP) created the tool Effectively Communicating 
Evaluation Findings, which is available on the OSEP IDEAs That Work website. It offers more information, tools, and 
strategies to help you effectively communicate your findings, with a focus on identifying key audiences, specifying 
their needs, and creating a communication plan tailored specifically to each audience. 


The Center for IDEA Early Childhood Data Systems (DaSy) developed a data visualization toolkit to help state staff 
effectively create and present data visuals. The toolkit includes design principles, data considerations, accessibility 
tips, general how-tos, examples, and sample tools related to a number of topics such as dashboards, charts, 
presentations, and data tables, and is located at http://dasycenter.org/data-visualization-Toolkit/. 


The Center for Parent Information and Resources (CPIR) offers a variety of resources for parents of students with 
disabilities, including a collection of resources that share lessons learned about effective dissemination. Resources 
include a dissemination self-inventory tool to assess and improve dissemination plans, a tip sheet for writing in plain 
language, and steps for crafting content and disseminating information through various media. 
http://www.parentcenterhub.org/dissemination 


As you can see, collecting high-quality parent involvement data and reporting it correctly and effectively to OSEP and 
other key stakeholders requires planning, coordination, and collaboration. In this toolkit we have walked you through 
the process of getting Ready to conduct your data collection, given you strategies to help ensure that your data 
collection Runs smoothly, pointed out important considerations to help you Reflect on and analyze your data, and 
offered suggestions for how you can communicate your findings to Reach important audiences for your data. We have 
offered links to other resources that may be helpful and presented examples of what is happening across the states. 


In the Deeper Dive that follows, you will learn about how to work with third-party contractors; involve stakeholders in 
your parent involvement data collection, analysis, and reporting activities; and modify or develop a new survey. 
Additional Resources offer important information about sampling and about the NCSEAM scale. Finally, in the online 
toolkit, we offer two examples of surveys that illustrate principles of good survey design. 
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DEEPER DIVE: What should we do if we want to work with 
a third-party contractor? 


If you want a Parent Center to take an 


If you or your staff do not have the expertise or time to collect, analyze, and 
active role in your parent involvement 


report your state’s parent involvement data, you may consider engaging third- deracollectionactivitices yeurnad 
party contractors—such as external evaluators, universities, federally funded heed Gveantract Rees ference 
Parent Training and Information Centers (herein referred to as Parent Centers), go beyond their typical IDEA-funded 
or consultants—who have the knowledge and skills to assist. You can engage grant activities. 

contractors in a single aspect or in multiple aspects of the work. 


In this section you will learn about how you can 
e Determine how the third-party contractor can help with your parent involvement data collection, analysis, and 
reporting activities 
e Bedeliberate as you go about finding and hiring a contractor 
e Take steps to ensure that the working relationship will be successful 
Determine how the third-party contractor can help 
If you are considering engaging a third-party contractor, begin by asking yourself (or others on your team), “What 
aspects of the work would benefit by bringing in an external professional?” 
Here are some important questions to get you started thinking about what kind of help you might need: 
e Does our staff have knowledge and skills in instrument design, sampling, item development, data collection, 
data analysis, data use, and reporting? 


e Dothe staff with the necessary skills have time for planning and conducting data collection and analysis, or are 
they assigned to other activities? 


e Dowe have available support personnel for stuffing envelopes, doing and tracking mailings, programming web 
surveys, and following up with nonresponders? 


e Howmuch of a third-party contractor’s time will our budget support??? 


Keep in mind that timing is important when working with third-party contractors. Contractors brought on when the 
state team is planning for data collection can provide input and guidance early on in the process, which will result in 
better data collection and analysis and, ultimately, better results. So, be sure to think about involving third-party 
contractors as early as possible to ensure they can provide you the help you need! 


Be deliberate about finding and hiring a third-party contractor 


There are a number of steps to take once you have decided to hire a third-party contractor. First, depending on your 
state’s specific requirements, you may need to develop a Request for Proposals (RFP) that includes 


e Adescription of the data collection instruments and methods 


e The scope of the contractor’s work 


39Carlson, E., and D’Agostino, A. (2015, July 22). Three-Part Webinar Series on Customer Survey Development. [Online Webinar.] Retrieved from 


https://www.osepideasthatwork.org/webinar-series/three-part-webinar-series-customer-survey-development. 
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e = Allist of all reports and other products the contractor will deliver and deadlines for those deliverables 


e Proposal requirements 
ee Important questions to ask... 


e What roles and responsibilities will state staff have in 


e The RFP timeline (including deadlines for submitting the data collection and analysis process (if any)? 


questions, submitting proposals, and making award e What, if any, types of instrumentation or forms need 
decisions)” to be identified or developed? Who will create and 


pilot test the new instruments? 


e Proposal evaluation criteria 


Or your state may already have an RFP template you can use e How will data collectors and data entry staff be 

or modify. trained? What materials, if any, need to be 
developed? Have all data collectors and data entry 

After you have your RFP, you should distribute the RFP to staff received training in the protection of human 

professional organizations, research and consulting firms, and subjects? 

local colleges and universities, as well as through relevant e What is the timeline for data collection? 

websites and listservs. e How will staff enter data into a database and verify it 
for accuracy? Where will data be stored? 

After receiving proposals, assess each contractor's e What security protocols will staff develop to maintain 


data confidentiality? Who will have access to the data 
after they have been entered? 

e Who are the proposed staff (with full-time equivalent 
and qualifications)? 

e At what points along the way will the contractor 
provide updates to the state and share the data? 


qualifications and experience related to the scope of work and 
be sure to check references. Select the most qualified and 
experienced contractor, taking into account the budget, and 
prepare a contract that specifies the expected responsibilities 
of the contractor. For specific information about finding and 


hiring third-party evaluators, see the Guidelines for Workin e When will the contractor deliver the final Indicator B8 
with Third-Party Evaluators available on OSEP’s IDEAs That data? How will the state ensure data are available in 
Work website. time to report in the annual SPP/APR? 


A Note on Working With Third-Party Contractors 

There are a number of benefits and limitations to working with a third-party contractor. Third-party contractors bring in technical 
expertise, credibility, and objectivity. They also take on the responsibility of completing key tasks, allowing state and district staff 
to focus on other priorities. However, using third-party contractors may mean additional costs. Also, even when third-party 
contractors have a substantial role in data collection, you will have important decisions to make about defining their scope of work 
and monitoring and managing the process. 


Take steps to develop a strong working relationship 


While it is important for the third-party contractor to maintain objectivity, the state and the contractor can develop a 
strong working relationship that facilitates completion of key tasks. The working relationship may be especially 
important when or if challenges arise. 


Two strategies for establishing a strong working relationship with third-party contractors are 


e Setting reasonable goals and expectations 
e Defining decisionmaking roles and responsibilities** 


It is important to recognize that hiring a third-party contractor does not eliminate the need for internal staff to spend 
time on collecting and reporting parent involvement data! Someone in the agency will need to maintain regular 
communication with the contractor and track the contractor’s progress to ensure they are completing necessary events 
and tasks on schedule and to expectations. 


Remember that ultimate responsibility for reporting high-quality parent involvement data still rests with the state. 


40 Heinemeier et al. 2014. 
41 Heinemeier et al. 2014. 
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DEEPER DIVE: How can my state involve stakeholders to 
improve the quality and usefulness of our parent 
involvement data? 


Many states seek input, recommendations, and assistance from a variety of 


. : : : In the context of Indicator B8, 
stakeholders on the collection and reporting of their parent involvement data. 


stakeholders can include parents and 
family members, OSEP-funded Parent 
Involving stakeholders is critical for Training and Information Centers 


(PTIs) or Community Parent Resource 
e Ensuring the data collection activities are relevant to and understandable Centers (CPRCs)—referred to as 
by parents Parent Centers—parent support or 
advocacy organizations, state 
advisory panel members, Parent 


e Ensuring accurate interpretation of data Teacher Association members, and 
local school district personnel. 


e Gaining parents’ buy-in and trust during data collection 


e Building positive relationships to get parent buy-in 
Determine which stakeholders to include 


When deciding to engage stakeholders, first think about which stakeholders to include. Depending on what you might 
want them to do, consider including parents and family members, parent support or advocacy and resource 
organizations, state advisory panel members, Parent Teacher Association (PTA) representatives, and local school district 
personnel. 


IDEA-funded Parent Centers are crucial resources for states to include throughout all phases of the process! 


e Parent Centers provide information, training, and support to parents, families, and guardians about their child’s 
disability; early intervention for babies and toddlers; school services for school-aged children; therapy; 
information on local policies; transportation; and much more. 


e Parent Centers are also key stakeholders in states’ program improvement efforts. 


e Many Parent Centers have staff who are experienced with data collection and outreach to diverse families and 
underserved populations, and who understand the importance of establishing trusting relationships with 
parents. 


Collaboration between states and Parent Centers is, therefore, critical to obtaining an accurate picture of a state’s level 
of parental involvement. Importantly, by working with Parent Centers, states can avoid possible barriers and facilitate 
collection of valuable parent data. 

Be clear about stakeholders’ roles and make it easy for them to contribute 

When engaging stakeholders, decide whether the group will have decisionmaking authority or will function in more of 


an advisory capacity, and be clear with them about their expected role. 


Keep in mind that many stakeholders may not be able to participate in person during regular school hours. Take steps to 
facilitate their participation, such as 


e Planning events or activities adjusted to the needs of parents who may not be able to participate in person or 
who cannot make an ongoing commitment 


e Holding online forums to request input and ideas on planned data collections or to get reactions to the results 
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e Developing briefs or holding and recording webinars related to the B8 data and then posting them on the state 


website 


Get stakeholder input on instrument selection and design 


Whether your state is using existing data collection instruments (such as surveys with or without modifications) or 
developing new ones, you should give stakeholders an opportunity to provide input. To ensure you will get accurate 
data that you can use to answer questions of interest to agency staff and key stakeholders, ask stakeholders to 


e Provide input on how to define parent involvement 


e Choose which language(s) to use to administer the survey 


e Give feedback on the overall content and on individual items to ensure the survey is easily understood, can be 
interpreted as intended by parents, and includes appropriate questions given the particular state context 


e Review the survey periodically (e.g., annually) to make recommendations for changes that are needed 


immediately or that may be helpful for future data collections 
Involve stakeholders in collecting data 


Stakeholders can also provide valuable support to data collection activities. For 
example, 


e Local districts can develop their own procedures to distribute the survey 
(within state guidelines), and send reminders to parents to complete 
the survey 


e Parent Centers, parent advocacy groups, and parent resource 
organizations can 


— Directly distribute surveys (e.g., by providing access through the 
Parent Center websites or mailing out the surveys) and include their 
branding on the materials 


— Encourage parent participation through their regular 
communications during meetings and events, email reminders, or 
messages about the survey posted on the Parent Center website 
and in social media outlets (e.g., Facebook, Twitter) 
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Oklahoma partners with the state’s 
Parent Center to administer the Part B 
Parent Survey. The survey is available 
online, on the Parent Center’s 
website, but parents can complete the 
survey by paper or by phone if 
needed. Parents and districts can 
contact the Parent Center to access 
those alternative methods. The 
website also hosts additional materials 
for parent and district use, including a 
survey brochure, frequently asked 
questions, and contact information if 
parents have questions. The Parent 
Center distributes hundreds of 
brochures to every district annually 
for IEP teams to share with parents at 
meetings. These brochures detail all 
the ways in which parents can 
respond to the survey. Special 
Education Services’ state personnel 
and the Parent Center developed the 
brochures. 
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Engage stakeholders as you reflect on the data collected 


Just as with the data collection, engaging stakeholders in data analysis is essential. Different stakeholder groups will 
bring unique perspectives to understanding the findings, so it is a good idea to include a variety of different stakeholder 
groups. The stakeholders involved during data analysis may be the same, or similar to, those engaged in collecting data, 


and stakeholders should be involved throughout the data analysis process. 
Here are some ways stakeholders can help as you reflect on your data: 


e Inthe beginning stages of data analysis—when you are doing initial 
calculations of response rates and checks for representativeness— 
stakeholders may be able to identify if parents in a particular 


geographic region or parents of students from certain subgroups need 


to be targeted to encourage them to participate in ongoing data 
collection activities 


e Once initial survey results are available, states can engage stakeholders 


in discussions that will 
— Promote the exchange of ideas 
— Increase or improve partnerships with stakeholders 


— Provide a more nuanced look at the data 


Involving your state’s PTI and 
CPRC during data analysis can 


be particularly valuable! PTis and 
CPRCs (Parent Centers) have a 
comprehensive picture of statewide, 
regional, and national parent and 
family needs across disability types. As 
part of a nationwide network, PTIs 
and CPRCs have access to a wealth of 
knowledge and resources that can be 
of benefit during analysis. 


e As data analysis proceeds, stakeholders can provide feedback to prompt further analysis that may help you 


better understand what the survey results mean 


e Stakeholders also may be able to help explain possible patterns of results and suggest strategies to address 


areas of concern 


e Stakeholder input on improving survey distribution in future years can be helpful to ensuring that ongoing data 


collection activities are successful 


Get help from stakeholders on how to best use and share 
findings 


Getting stakeholders involved in data dissemination and use goes beyond 
just having them review and discuss results of data collection activities. For 
example, stakeholder feedback on parent survey data may contribute to a 
larger decisionmaking process around overall program improvement and 
priorities. Allowing them to provide suggestions for how to use the results 
of the data collection to inform decisionmaking can lead to improvements 
in current and future programs. 


Stakeholder involvement in the dissemination and use of parent 
involvement data is key to improving services and results for students with 
disabilities and their parents. Involving stakeholders in this way can 


e Offer state agencies additional expertise and guidance to identify 
and address issues with parent involvement 


e Encourage families, communities, and schools to share important 
ideas for strategies to further improve parent involvement 


e Build positive relationships between the state, districts, schools, 
families, and communities 
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New Hampshire (NH) Connections was a 
project of the Parent Information Center 
and funded by the New Hampshire 
Department of Education, Bureau of 
Special Education, to build the capacity of 
families of children with disabilities and 
school staff to work together in 
partnership. The 2013-14 New Hampshire 
Parent Involvement Survey in Special 
Education Statewide Summary 
(https://www.education.nh.gov/ 
instruction/special_ed/documents/2013 
2014 nh _ parent involvement survey in 
sped.pdf) describes specific ways in which 
NH Connections assisted districts in using 
data from the parent involvement survey 
to improve family-school partnerships in 
special education and outcomes for 
students with disabilities. 
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e Make more people aware of the nature and extent of parent involvement in the education of children with 
disabilities 


Box 8 presents some resources that you can use to engage and work with stakeholders. 


Box 8. Resources for engaging and working with stakeholders 


Project Forum, a federally funded project at the National Association of State Directors of Special Education (NASDSE), 
conducted an analysis to identify features of successful collaborative partnerships between states and OSEP-funded PTIs to 
improve outcomes for Indicator B8. The policy brief describes specific examples of collaboration between four states and the PTIs 
in the areas of professional development and technical assistance, development of training and/or guidance materials, 
dissemination of information, parent outreach, and dispute resolution (http://nasdse.org/DesktopModules/DNNspot- 


Store/ProductFiles/21 c9ec21a1-3101-4d8f-bc7e-de63e88225df.pdf). 


Serving on Groups is a guidebook to support families in serving on decisionmaking groups, but the information and structure of 
the guidebook can support anyone who is currently serving, or wants to serve, on a decisionmaking group. Section 6, 
Understanding Data as Information, focuses on using and understanding data. http://www.servingongroups.org/ 


Leading by Convening is a framework for creating learning partnerships across stakeholders. It includes steps for coalescing 
around issues, ensuring relevant participation among group members and effectively working together as a group. 
http://www. ideapartnership.org/documents/NovUploads/Blueprint%20USB/NASDSE%20Leading%20by%20Convening%20Book. pdf, 


The Center for Parent Information and Resources (CPIR) offers a variety of resources for parents of students with disabilities. 
One webinar that may be of particular interest is Using Data for Collaboration and Advocacy. 


http://www. parentcenterhub.org/webinar-using-data-for-collaboration-and-advocacy/ 


The PACER Center offers assistance to families, workshops, materials for parents and professionals, and leadership in securing a 
free and appropriate public education for all children. http://www.pacer.org/ 
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DEEPER DIVE: How dol! develop a new survey or modify an 
existing one? 


Currently, all states are conducting surveys to gather their Indicator B8 data. Approximately 40 percent of states have 
reported using a state-developed survey, and an additional 11 percent have reported using a modified version of the 
National Center for Special Education Accountability Monitoring (NCSEAM) scale“? or the Early Childhood Outcomes 
(ECO) survey.“ Given the widespread interest among states to use either their own state-developed survey or to modify 
an existing survey, it is essential that state staff and other 
individuals who are involved with developing or modifying and Determine what 
administering states’ parent involvement surveys are aware of to measure 
important principles of survey design. 


Correctly developing or modifying a survey can be a complex 
and lengthy process. There are several important steps involved 
and, at each step, there are multiple decisions that require 
careful consideration if you want to ensure that your state’s 
survey produces high-quality parent involvement data. Rather 
than presenting an extensive discussion of survey methodology, 
in this section, we highlight some important recommendations 
to follow when creating a new survey. 


Create items 


Design survey 


e Determine what you will measure Pilot survey 
e Create survey items 


e Design the survey to maximize responses 


e Pilot the survey Revise survey 


e Revise the survey based on the pilot 


The principles are generally the same for modifying an existing survey. There are many good books about the process of 
developing surveys” and there are webinars related to customer surveys available on the Office of Special Educations 
Programs (OSEP) IDEAs That Work website. We urge you to review these for more detailed information about survey 
development. You can also consult your IDC State Liaison for more information. 


42 “Schools’ Efforts to Partner with Parents Scale” (SEPPS) was developed by the National Center for Special Education Accountability Monitoring (NCSEAM) specifically 
to meet the requirements of Indicator B8. The scale is commonly referred to as the “NCSEAM.” 


43 The Family Outcomes Survey was developed by the Early Childhood Outcomes (ECO) Center. This survey is commonly referred to as the “ECO.” 
44 For example, Czaja and Blair, 2005; Dillman, Smyth, and Christian, 2009; Groves, Flower, Couper, et al., 2004; and Harkness, Braun, Edwards, et al., 2010. 
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Determine what you will measure through your survey 


The first step in designing a high-quality survey is to determine what it is you want to measure. Indicator B8 asks states 
to report the “percent of parents with a child receiving special education services who report that schools facilitated 
parent involvement as a means of improving services and results for children with disabilities (20 U.S.C. §1416(a)(3)(A)). 
There are two important concepts included in this indicator: 


n” 


e What “parent involvement” is 


e What it means for schools to “facilitate parent involvement as a means of improving services and results for 
children with disabilities” 


To be able to collect information on these concepts, you must develop a state-specific definition to “operationalize” 
parent involvement. (Note: We will only focus on the concept of “parent involvement” in this section, but the process of 
defining and operationalizing other key concepts for your survey is virtually the same.) 


Define parent involvement 


As you think about what you want to measure through your How Can Stakeholders 
survey, you must decide how to define parent involvement, Ip? 

thinking carefully and identifying the factors that indicate whether He Pp: 

and to what degree a parent could be considered to be involved. Engaging stakeholders—including parents, 
For example, does parent involvement include only attending Parent Center staff, state advisory panel 
individualized education program (IEP) meetings, or does it also members, community representatives, 
require working with school staff to develop their child’s IEP? Is district and school staff, state staff, and 
there a certain level of involvement that your state is looking for? individuals representing other agencies 


and organizations that are engaged with 
parents—in the process of defining parent 
involvement can ensure that your 
definition of parent involvement is 
appropriate given your state’s unique 
context, priorities, and activities, and that 
the results of the survey will ultimately be 
relevant to stakeholders. 


Defining key concepts and terms is one of the most important 
tasks when developing or modifying a survey because your 
definitions will determine what you are trying to measure, 
endorse a common understanding among all individuals involved 
in the data collection process, and provide an organizing structure 
to help guide your efforts. 


As you work, you may want to engage stakeholders and 

investigate models of parent involvement to see if any resonate with your state’s context. Keep in mind that you can 
modify existing models to fit your state’s own characteristics and needs (see Box 9 for examples of parent involvement 
frameworks). It is important, however, that you ensure that your state’s definition is consistent with the intent of 
Indicator B8. 
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Box 9. Examples of parent involvement frameworks 


ES’s Toolkit of Resources for Engaging Families and the Community as Partners in Education: This toolkit 
provides resources to help educators build relationships with families and community members, support family 
well-being and relationships, and foster students’ learning and development. The toolkit is broken into four parts: 
1) building an understanding of family and community engagement, 2) building a cultural bridge, 3) building 
trusting relationships with families and community through effective communication, and 4) engaging families and 
community members in data conversations. 


The Dual Capacity-Building Framework for Family-School Partnerships: This framework supports families, 
schools, districts, and states in building capacity for student achievement and school improvement. The 
framework outlines a process that schools and districts can use to build the type of effective family engagement 
that will make schools the centers of their communities. 


The Head Start Parent, Family, and Community Engagement Framework: This framework helps providers assess 
and track progress across key indicators of effective family engagement to support children’s learning and 
development. 


Family Engagement Framework: This framework, developed for districts in California, describes expectations and 
implementation strategies for integrated family engagement within state educational programs and provides 
guidance for planning, implementing, and evaluating strategies for effective family engagement to support 
student achievement. 


Epstein’s Framework for Six Types of Involvement: This framework includes sample practices, challenges, and 
results for six types of parent involvement: 1) parenting, 2) communicating, 3) volunteering, 4) learning at home, 
5) decision making, and 6) collaborating with community. 


Operationalize parent involvement 


Once your state has defined parent involvement, you will need to operationalize the definition, which means providing 
additional detail about your definition so that it is specific and can be measured. For example, say your definition of 
parent involvement includes parents working with school staff to develop their child’s IEP and attending IEP meetings. 
To operationalize the definition, you need to specify: 


e How many IEP meetings parents should attend 


o Any? 

o All? 

o Aspecific percentage (e.g., 80%)? 
o Other: ? 


e What it means for parents to “work with school staff to develop their child’s IEP” 
o Discussing plans for the IEP with school staff? 

Reviewing the draft IEP and providing feedback? 

Co-creating the IEP along with the school staff? 

Other: ? 


OOO 


You should operationalize all key concepts (such as “facilitating parent involvement as a means of improving services 
and results for children with disabilities”) that your state is measuring through your parent involvement survey. This will 
help you create survey items that align with your operational definition. 
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Here are some questions you might ask when thinking about how to 


: : F One way to operationalize your 
operationalize parent involvement: 


definition is to work to make it 


“ 45 
e What does parent involvement mean in the context of my state? SMART": 


e Specific and clearly stated 


e What does parent involvement look like in practice? 
e Measurable and based on data 


e Does parent involvement include elements of quantity (e.g., the e Attainable 
number of times parents attend meetings) as well as quality (e.g., the e Realistic/relevant 
extent to which parents participate during those meetings)? e Time-bound 


e How will we know whether parent involvement is happening? (i.e., what indicators can we use to measure 
parent involvement?) 


A key part of the process of operationalizing your definition is determining which indicators of parent involvement you 
can actually measure. It might be possible to devise a very specific operational definition, but if you cannot feasibly 
measure that indicator through your survey or other data collections, then the definition will not be helpful. 


Develop a framework for the survey 


Another important part of determining what you will measure is developing a framework for the survey that includes 


e Theitem topics (i.e., what you will ask) 
e The target population of interest (i.e., who you will ask) 
e The mode of administration (e.g., how you will ask them) 


Item Topics. In the REACH section of this toolkit, we discussed how you can go beyond Indicator B8 to make the most of 
your parent involvement data. You should include items in your survey that will enable you to answer the different 
questions your state has about parent involvement. However, while it may be tempting to have your survey cover all 
possible aspects of parent involvement, we strongly recommend that you focus on those questions that are the most 
important to your state. Focusing your survey on a smaller number of important questions will help you to conserve 
resources (e.g., it will save time if your team does not have to analyze extensive amounts of data), reduce respondent 
burden, and potentially increase your response rates. Further, if you plan to do additional data collection activities (e.g., 
observations or interviews), you can focus the activities on gathering more in-depth data, as needed. 


Target Population. As we discussed in the section on selecting the group who will participate in data collection, you need 
to decide if you want to focus your parent involvement data collection activities only on obtaining data from parents, or 
if you want to include additional stakeholders (e.g., district or school staff) as well. Think about who you want to survey 
and determine how you will select the survey sample. Keep in mind that if you are using more than one method to 
collect data (e.g., surveys and interviews), you might have different samples for each method. For more information on 


sampling, see the additional resources on sampling. 


Mode of Administration. In the section on creating data collection procedures and tools, we talked about two basic 
modes of survey administration: online surveys and paper surveys. See that section for important considerations related 
to the survey mode. 


45 This acronym was coined by George Doran for management purposes but has since been adapted and used by multiple authors in varied ways. 
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Create survey items 


Once you have operationalized parent involvement and created a framework for your survey, you need to create the 
items you will include in your survey. When creating survey items, always keep in mind the original purpose for your 
parent involvement data collection activities: What questions do you want to answer? 


When developing new survey or modifying existing survey items, be sure to follow these important principles: 


e Use clear and simple language to facilitate understanding 

e Select the correct item format to get the data you desire 

e Offer good response options to help respondents give accurate answers 
Use clear and simple language 


Use clear and simple language when writing survey items to increase the likelihood that the people completing your 
survey will understand how to answer each question. Figure 12 illustrates the process respondents follow when 
answering a survey question, and shows that if respondents cannot understand the question, or if the answer 
respondents want to give is not one of the response options, they will not be able to respond accurately. (Or they may 
not even try to answer at all!)*° 


Figure 12. The process of answering a survey question 


Respondent Respondent Respondent seiaainae Respondent is 
stores comprehends retrieves penne able to fit 
information question information answer with 
question response 
options 


ee rer rr 


Table 26 presents some characteristics of a good survey item, with examples of questions that ensure that the items are 
written in a way that facilitates (or does not facilitate) understanding. 


46 Fiore et al. 2012. 
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Table 26. Characteristics of a good survey item, with examples of not good and 


good items 


Characteristic 
Uses clear 
terminology and 
plain language 


Example: Not good 


How has your active engagement 
enhanced your child’s academic 
or social-emotional 
performance? 


In what ways does your involvement help your child succeed in 
school? 


Example: Good 


Only asks one 
question at a time 
(avoids double- 
barreled questions) 


To what extent do you think the 
afterschool workshops and 
Parent Nights have increased 
your access to information about 
how to be more involved in your 
child’s education? 


e To what extent do you think the afterschool workshops have 
increased your access to information about how to be more 
involved in your child’s education? 

e To what extent do you think the Parent Nights have increased 
your access to information about how to be more involved in 
your child’s education? 


Contains a clear 


Have you met with your child’s 


Have you met with your child’s teacher to discuss your child’s 


threshold for teacher? progress achieving his or her IEP goals? 

answering “yes” 

Provides a How often do you receive How many times in the past month have you received 

timeframe information from the Parent information from the Parent Information Center? 
Information Center? 

Provides a How many times did you attend How many times did you attend IEP meetings last year? 

timeframe IEP meetings? 

appropriate to the 

topic 


Gives exhaustive 
and mutually 
exclusive response 
options 


In which situations are you given 
opportunities to provide input on 
your child’s education? 
e During meetings with 
school staff 
e During informal 
communications with 
school staff 


In what situations are you given opportunities to provide input 
on your child’s education? 


During IEP meetings 

During other regular in-person meetings with your child’s 
teacher 

During other meetings with related services providers 
assigned to work with your child 

Through emails or other written communications with 
your child’s teacher or related services providers 
Through regularly scheduled phone calls 

Through unscheduled phone calls imitated by you or by 
staff 

Through informal communications with staff, such as 
chats before school 


Avoid double- 
negative questions 


It is not an insignificant burden 
for me to attend IEP/IFSP 
meetings. 


Please indicate the degree to which you agree with this 
statement: It is a significant burden for me to attend my child’s 
IEP/ISFP meeting. 


Other ways you can work to use clear and simple language include 


e Keeping questions brief 


e Ensuring that the items are written using language no higher than an eighth grade reading level 


e Avoiding potentially unfamiliar terms and abbreviations whenever possible 


e Defining any unfamiliar terms or abbreviations 
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Select the correct item format and type 


Another part of creating a good survey item is to select the correct item format. Table 27 presents three types of item 


formats.*” 


Table 27. Item formats 


Item format 
Closed, or structured, 
items — Provide 
predetermined 
response options 


Works well when 
You have many respondents. 
You have (or can create) a list of possible 
response options. 
You want to expend fewer resources 
collecting and analyzing data. 
You need to calculate reliability of the data. 
You would like to conduct quantitative 
analyses of the data. 


Example 


In the past year, how many times have you met 
with your child’s teacher about your child’s 
academic progress? (Select one) 


a. 
b. 
c. 
d. 
e. 


Never 

1-3 times 

4-6 times 

7-9 times 

10 or more times 


Open, or 
unstructured, items — 
Allow respondents to 
provide answers in 
their own words 


You have a limited number of respondents 
(or you have extensive resources and staff to 
collect and analyze the data). 

You do not have (or do not know how you 
would create) a list of possible response 
options. 

You would like unthought-of categories to 
emerge. 

You want spontaneous, or in-depth, 
responses. 

You are exploring response options for future 
surveys. 

You have skilled qualitative analysts who 
have time to analyze the data. 


Over the past year, what would you say has 
been the most important thing your child’s 
school has done to facilitate your involvement in 
your child’s education? (Write your answer) 


Mixed format items — 
Combine elements of 
closed and open 
formats 


You would like to have the advantages of a 
closed format question (e.g., having the 
respondent pick from specific response 
options, minimizing costs) while giving 
respondents an opportunity to either enter 
their own response or elaborate on their 
answer. 


In the past year, in what ways has your child’s 
school made it easier for you to participate in 
your child’s education? (Select all that apply) 


a. 


Communicated with you regularly about 
your child’s progress on IEP goals. 

Given you choices about the services 
provided to meet your child’s needs. 
Offered you a variety of ways to 
communicate with teachers/school staff. 
Provided information on the options you 
have if you disagree with a decision of the 
school. 

Other, specify 


47 Carlson and D’Agostino, 2015. 
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Additionally, you can choose from a number of types of closed items, as shown in Table 28. 


Table 28. Types of Closed Survey Items 


Item Type Description Example 
Dichotomous | Respondents choose one of | Has your child received special education services during the current school 
two response options year? (Select one) 
1. Yes 
2. No 
Categorical Respondents choose one What grade is your child in? (Select one) 
response from among 1. under K-2 
multiple categories [Note: 2. 3-5 
These are treated as 3. 6-8 
nominal (or categorical) 4. 9-12 
data in analyses]. 
Forced Respondents choose one of | | receive communications from my child’s school... (Select one) 
choice: several response options 1. Not often enough 
Single 2. Often enough 
response 3. Too often 
Forced Respondents choose more How do you communicate with your child’s service providers? (Select all that 
choice: than one of several apply) 
Multiple response options 1. In-person 
response 2. Phone 
3. Email 
4. Other 
[Note: The “Other” response could also include a space for respondents to 
write in their answer, making it a mixed format item, as shown below.] 
Ranking Respondents rank items in Please rank the following methods of communication from most effective to 
a list in order of preference | least effective, with a 1 = least effective and 5 = most effective 
or importance. In-person meeting 
Phone call 
Text message 
Email 
Other (Specify): 
Rating Respondents select a single | Please indicate your level of agreement with the following: | feel involved in my 
rating along a continuum. child’s special education services. 
Strongly disagree Disagree Agree Strongly agree 
LI LI a LI 
[Note: This item does not offer respondents an option to enter a “neutral” 
response. This might be beneficial when you want to force respondents to give 
an opinion about a topic. However, it might discourage respondents who really 
do not have an opinion from responding to the question.] 
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Grid Questions. If you have a number of items that share a common set of response options (e.g., Yes/No, 
Strongly Agree to Strongly Disagree), you can group them into a grid or matrix, as illustrated in Table 29 and 


30. 


Table 29. Grid question example 1 


For each of the questions below, please check the box that corresponds to your 
answer. 


1. 


A support network for parents of students with disabilities is available to me 
through my school district or other sources. 


Yes 


No 


L 


Unsure 


L 


lam involved in a support network for parents of students with disabilities 
available through my school district or other sources. 


L 


L] 


Table 30. Grid questions example 2 


Please check the box that indicates to what extent you agree or disagree with 
each of the following statements. 


My child’s school: 


Strongly 
Disgree 


Disagree 


Agree 


Strongly 
Agree 


school. 


L 


L 


1. Communicates with me regularly about my child’s progress on IEP goals. [| [ ] [| [ ] 
2. Gives me choices about the services the school provided to meet my child’s 

eae LI LJ } OW} 
3. Offers mea variety of ways to communicate with teachers/school staff. [| [ ] [| [ ] 
4. Provides information on the options | have if | disagree with a decision of the 


L 


L 


Grid questions can reduce the amount of time it takes for respondents to complete a set of items.*® 4? However, it is 
important to structure these items well and make clear how you want respondents to complete the questions. Consider 
these two examples: 


In Table 29, the response option “Unsure” was not offered for item 2, because the survey designers decided that 
parents should know whether or not they are involved in a support network for parents of students with 
disabilities. If the “Unsure” response option had been available for item 2, then the data the state obtained from 
that item might have been unclear, since the state would not have known if the parents were responding 
“Unsure” because they weren’t involved in a network or if they were unsure if the network was “available 


through my school district or other sources.” 


In Table 30, the instructions tell respondents to check the box that indicates to what extent they “agree or 
disagree with each of the following statements.” If the word each had not been included in the instructions, 
then parents might have thought they did not have to respond to each of the items. Similar instructions were 
included in Table 29 (i.e., “For each of the questions below”) to ensure that parents answered both items. 


48 Couper, Traugott, and Lamias, 2001. 


49 Tourangeau, Couper, and Conrad, 2004. 
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When using grid questions, keep in mind the following suggestions: 


e Keep grid questions simple 


e Minimize the number of items in the grid 


e Provide direct instructions as well as visual clues to guide parents on how to respond 


e Repeat headers of grid questions on any new pages 


Since different item formats and types work best to gather different types of data, when choosing item formats and 


types, think carefully about 


e The questions you want to answer 


e The resources you have available to design and administer the survey and collect and analyze the data 


e The analyses you want to conduct 


This will help you choose the right format/type to obtain the data you need. Talk with an experienced survey developer, 
with your evaluator, or with your IDC State Liaison if you have questions. 


Offer good response options 


It is important to carefully consider how you will present response 
options for closed (and mixed format) items because the types of 
options available will affect respondents’ willingness to respond to 
the item and shape the conclusions you can draw from your data. 
For example, if respondents have a “neutral” opinion about some 
experience but do not see “neutral/no opinion/neither agree nor 
disagree” as a response option, they may skip the survey item. 


When developing response options for your parent involvement 
survey, it may be a good idea to talk with key stakeholders to get 
their input. They are in a good position to know what possible 
responses might be appropriate for a particular question, given 
the state context. 


Here are some important general tips to remember when creating 
your response options: 


How Can Stakeholders 
Help? 


Stakeholders can be a good source of 
information during survey development! 
As you think about possible survey items 
and corresponding response options, ask 
stakeholders to offer suggestions and give 
feedback to ensure respondents will be 
able to understand and interpret survey 
items as intended and the survey includes 
appropriate questions and response 
options given the particular state context. 


e Link response options back to the questions your state wants to answer. Be sure that the response options you 
provide will actually produce the data you are trying to collect. Consider this example: 


— Your state wants to know whether schools are actually using a specific strategy to facilitate parent 


involvement that was taught in a statewide professional development workshop. The strategy is 


providing families with a planning guide for IEP meetings to facilitate two-way planning and 
communication. Look at the two examples for Question 1 that follow and think about which is more 
likely to produce the data your state needs to answer this question and why. 
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Question 1, Example A Question 1, Example B 

In what ways does your Please indicate whether your school uses any of the following strategies to facilitate 

school facilitate your your involvement in your child’s education: 

involvement in your a. Provide you with a planning guide for IEP meetings. 

child’s education? Please b. Communicate with you regularly about your child’s progress on IEP goals. 

write your response: c. Give you choices about the services the school provides to meet your child’s 
needs. 


d. Offer you a variety of ways to communicate with teachers/school staff. 
e. Provide information on the options you have if you disagree with a decision of 
the school. 


Example B has clear advantages because it specifically lists the strategies for increasing parent involvement 
taught in the state PD workshop. This ensures that parents provide responses related to the use of those specific 
strategies. In contrast, Example A allows parents to decide what to talk about in their response, which may or 
may not provide data related to what the state would like to know. 


e Make sure response options are clear and mutually exclusive. Do not offer response options that overlap or 
that different respondents might interpret differently. Consider the three examples for Question 2 that follow. 
Which of the examples offers clear and mutually exclusive response options? 


Question 2, Example A Question 2, Example B Question 2, Example C 
How often do you communicate How often do you communicate How often do you communicate 
with your child’s teacher about with your child’s teacher about with your child’s teacher about 
your child’s IEP progress and goals? your child’s IEP progress and goals? —_ your child’s IEP progress and 

a. Sometimes a. 1-2 timesa year goals? 

b. Often b. 3-4 times a year a. 1-2 times a year 

c. Frequently c. 4ormore times a year b. 3-4times a year 


c. 5ormore times a year 


Example A is vague and response options b and c are overlapping terms. Example B provides clearer response 
options, but choices b and c overlap. Example C responses are clear and mutually exclusive. 


e List all of the potential answers to a particular question in your response options. When writing questions, be 
sure to include the full range of responses that might be possible. For example, if you are asking parents to 
report on their ethnicity, use all of the federal racial/ethnic group categories. See the two examples for 
Question 3 that follow. Which is the better choice and why? 


Question 3, Example A Question 3, Example B 
Which method do you most prefer that your child’s Which method do you most prefer that your child’s 
school use to notify you about upcoming IEP school use to notify you about upcoming IEP 
meetings? meetings? 
a. Mail a. Mail 
b. Phone call b. Phone call 
c. Email c. Email 
d. Text message 
e. School website 
f. Other (Specify): 


Example B is the better choice. Example A gives parents very few response options—what happens if parents 
prefer another type of communication? Example B provides more response options and also includes an “Other 
(Specify):” option with a space for parents to write-in a response. This gives parents even more ways to tell the 
state how they prefer to receive communications. However, be careful when including the “Other (Specify):” 


option, or you will end up with large amounts of qualitative data that you will then need to analyze. 
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Finally, for questions that might not apply to all respondents, you may want to include a “not applicable” 
response option. Be careful when offering a “not applicable” response, however, because it might encourage 
parents to choose that response to avoid having to provide an answer to the question. 


e Use fully labeled scales.*° Labelling all of the points on the scale helps to ensure that everyone has a similar 
understanding of the response options. For example, you should label every point along the scale rather than 
labeling only certain points of the scale (e.g., the endpoints) and using numbers or dashes to represent the other 
points on the scale. Consider the examples presented that follow. How might parents’ responses differ 
depending on which response scale was provided? How should the state interpret the different responses for 
Question 4, Example A? 


Question 4, Very Very 
Example A Unsatisfied a 2 3 4 Satisfied 
Question 4, Very Somewhat A Little A Little Somewhat Very 
Example B Unsatisfied Unsatisfied Unsatisfied Satisfied Satisfied Satisfied 
LI L] LI LI LI LI 


e Present response options in the order that respondents would generally expect. Consider the two examples 
for Question 5 that follow. Since many parents only meet annually with their child’s IEP team, it might make 
more sense to present the response options as shown in Example B. 


Question 5, Example A Question 5, Example B 
How often do you meet with the IEP team to planand How often do you meet with the IEP team to plan 
discuss your child’s educational progress? and discuss your child’s educational progress? 

a. 5ormore times a year a. 1-2 times a year 

b. 3-4 times a year b. 3-4 times a year 

c. 1-2 times a year c. 5ormore times a year 


e Additionally, whenever possible, we recommend that you present scales horizontally, moving from left to right, 
as shown in the scale that follows. 


Strongly Disagree Disagree Neutral Agree Strongly Agree 


L L a] L L 


5° Toepoel and Dillman, 2008. 
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Design the survey to maximize responses 
| 


While it is essential that you have good survey items that actually measure the concepts you are trying to measure, it is 

equally important that you pay attention to the overall design of your survey to maximize the overall response rate and 

the item response rates. The way you design your survey greatly influences whether respondents are willing and able to 
complete all of the items in a way that gives complete and accurate data. For this reason, you must be purposeful when 
designing your survey. 


In the section on creating data collection procedures and tools to facilitate participation and track responses, we 
discussed how the data collection mode or delivery method can affect both data quality and response rate. We also 


pointed out several issues associated with using online and paper surveys. Check out that section if you haven’t already 
done so. Here, we point out some important considerations for designing online and paper surveys. Additionally, keep 
the general tips in Table 31 in mind as you create your survey! 


Table 31. General Tips for Designing Surveys 


Survey introduction 


Y¥ Include a title that respondents and others can easily understand and that describes the survey 
content. 
¥ Explain who you are and the purpose of the survey. 
¥ Mention the time commitment up front. 
Y Provide contact information, in case respondents have questions or feedback. 
Survey layout 


¥ Keep your survey as short as possible. 

v¥ Keep the survey instrument simple and uncluttered. 

v¥ Number survey items and pages. 

vY Provide enough space for people to write responses or comments to open-ended questions. 
Item order 


¥ Start with an easy or interesting question. 

Y Place the most important questions near the beginning of the survey, and the most difficult or sensitive 

items toward the end. 

Order items in a logical way and number them sequentially. 

Group items by topic and introduce each change in the topic covered. 

Y Use italics for instructions and bold or underline for emphasis. Apply the use of these text styles 
consistently throughout the survey, so respondents can learn the meaning associated with each text 
style. 


AVA 


Special considerations for designing online surveys 


If you are planning to use an online survey, keep in mind these design recommendations. 


e Use a paging design. There are two primary designs for displaying items in an online survey: scrolling and 
paging. With the scrolling design, the survey presents all of the questions in a single scrolling web page and data 
are not saved until the end of the survey. With the paging design, the survey presents each question on its own 
page and data are saved as each page is completed. We recommend using paging designs because they are 
associated with fewer missing items and shorter completion times. 
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e Include “Next” and “Previous” buttons on the bottom of each page. These buttons allow users to navigate to 
the various pages, as desired. 


e Consider whether to use progress monitors. Progress monitors are displays showing what percent of the survey 
has been completed at a given point. There is debate around whether to use progress monitors in paging 
designs. Many users find such progress monitors helpful, while others may get discouraged if they have 
completed many items and still are seeing that they have completed only a small part of the survey. By keeping 
the survey short, you can use progress monitors and avoid the risk of discouraging responses. 


e Use a neutral background color. The colors you choose will affect readability. Use a neutral background that is 
free of graphics that may be distracting. 


e Use a typeface and font size that permit comfortable reading. The typeface should be easily readable, and the 
font size should be large enough to permit comfortable reading. 


e Use headers and place important information at the top of the page. A header on each page of the survey can 
facilitate completion. Place the most important information at the top left section of the page. 


e __Left-justify question text. Left-justifying question text can help responders identify the beginning of each 
question more easily. 


e Use a survey program compatible with Section 508 compliance. Be sure that your online survey program has 
the capability to create surveys that are Section 508 compliant. 


Many online survey tools have the design features described previously built-in, so they make it relatively easy to create 
online surveys. Additionally, many online tools offer survey templates that you can start with, rather than having to 
begin from scratch. 


Another benefit of using online surveys is that they offer many advanced design features that are not available with 
paper surveys, such as 


e Automatic skip patterns (e.g., “rules” built into the survey that automatically move from one question to 
another question based upon how respondents answer) 


e Edit checks (e.g., automatic checking for out-of-range answers, such as when a parent enters the number 45 as a 
response to a question asking the age of his or her child) 


e Automatic “fills” or “prepopulation” of answers (e.g., automatically inserting information from prior answers 
into subsequent questions) 


When planning your survey, consider whether you would like to use design features such as those listed previously and 
start thinking about how you should structure your survey to make the most of those features. Box 10 lists some online 
survey tools that may be of interest to you. Please note that inclusion of this list does not indicate endorsement of any 

particular online survey tool by IDC or OSEP. 
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Box 10. Selected online survey tools 


Here are some online survey tools that your state might use. Some of these offer a variety of free options 
(usually with some sort of registration), while others require payment. Those that offer free options may have 
restrictions on the number of surveys that you may send out through the free service. Be sure to thoroughly 
investigate the different options to find an online tool that will work best for your state. In some cases, it might 
be best to hire a third-party contractor to develop and program an online survey for your state. 


Zoho Survey 

SurveyGizmo 

Checkbox Survey 

FluidSurveys 

SurveyMonkey 

WorldApp KeySurvey 
GetFeedback 

Outside Software eSurveysPro 
SoGoSurvey 

SurveyPlanet 


Special considerations for designing paper surveys 


If you are designing a paper survey, here are some tips to increase the likelihood that respondents will be willing and 
able to complete the survey and provide accurate responses. See the sample surveys in the Resources section of this 
toolkit for examples of how to put these tips into practice. 


e Design survey covers that look interesting and appeal to respondents. Front covers should include a clear title 
and sponsorship information. Back covers should thank respondents for their time and effort in responding, 
provide a place for them to add any additional comments, and provide a return address. 


e Present the survey in booklet format. You can create booklets of four 8.5” x 11” pages by printing your survey 
on 11” x 17” paper as shown in Figure 13. When folded, the booklets will fit into 9” x 12” envelopes for 
mailing.°? When using booklets, add or delete pages in units of four (i.e., a sheet of folded paper). 


51 Adapted from Dillman, Smyth, and Christian (2014) 


www.ideadata.org 88 


IDC Parent Involvement Toolkit 


Figure 13. Example of how to create a booklet using 11” x 17” paper 


I a 


ITEM 6 
ITEM 7 
11” : JTEM 8 


ITEM 9 
ITEM 10 


8.5” 


ITEM 6 
ITEM 7 11” 
ITEM 8 


ITEM 9 
ITEM 10 


Consider aligning questions in more than one column. Depending on how many questions you have and how 
complex your survey is, you may want to present your questions in one or two columns on the page. A single- 
column format, such as the example that follows, works better for longer or more complex questions. 


For each of the questions below, please check the box that corresponds to your answer. Yes No Unsure 


school district or other sources. 


1. A support network for parents of students with disabilities is available to me through my 


LJ 


my school district or other sources. 


2. lam involved in a support network for parents of students with disabilities available through Cc Cc a 


The two-column format that follows works well for relatively short questions. 


2. IEP meetings are scheduled at times that are 
convenient for me. Would you say,,,, 
CJ Strongly agree 
CJ Agree 
oO Neither agree nor disagree 
J Disagree 
C Strongly disagree 


3. | work together with the IEP team as an equal 
partner to develop my child’s IEP. Would you 
Says. 

CJ Strongly agree 

(J Agree 

oO Neither agree nor disagree 
(J Disagree 

CJ Strongly disagree 


7. All special education services identified in my 


child’s IEP haye been provided. Would you say... 


CL] Strongly agree 

(] Agree 

(J Neither agree nor disagree 
CL] Disagree 

CJ Strongly disagree 


lam comfortable asking questions and 
expressing concerns about my child to school 
staff. Would you say.,,. 

CL] Strongly agree 

LJ Agree 

LC] Neither agree nor disagree 

CL] Disagree 

C1 Strongly disagree 


Additionally, try to group related questions on one page or on two facing pages (if using a booklet format). 


Provide clear instructions for completing the survey, including instructions for how to follow skip patterns. 
Skip patterns are rules that tell respondents how to complete the survey items. Based on the way respondents 
answer particular items, skip patterns tell respondents to skip certain questions that they do not need to answer 
or continue with follow-up questions, as appropriate. There are several branching designs that can be used to 
present skip patterns, but one that works especially well is the “reverse and indentation method.” ** 


52 Ibid. 
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In the reverse and indentation method, the response category that will lead to a follow-up question is listed 
second so arrows can be placed to the right of response categories. The follow-up questions are visually set 
apart from the main pathway to create a hierarchy in the questions (see following example). This helps reduce 


skip errors.°? 
Q10. Is there a support network for parents of students with disabilities available through your school district or 


other sources? 
[_] No—> Go to Question 12. 


[| Yes 
1 


Q11. Are you involved in the support network for parents of students with disabilities? 


[| No 
[ ] Yes 


Q12. Does your child’s school provide information on the options you have if you disagree with a decision of the 


school? 


[ ] No 
[| Yes 


53 Gohring and Smyth 2013; Redline et al. 2003. 
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Box 11 provides some tips related to using web links in written communications about your data collection activities. 


Box 11. Using links to online surveys in printed communications 


Some states mail, or ask districts and schools to give to parents, printed communications inviting parents to 
complete an online survey with a web link to access the survey. These communications might be survey 
invitations or reminders such as letters, flyers, or postcards. While this can be a good strategy when states 
or districts do not have good email addresses for parents, you should follow these tips to ensure that 
parents can use the link to access your survey. 


Make sure the link is readily visible in the materials (e.g., use a large font and clear instructions 
such as “Enter this URL into your web browser to access the survey: 


http://www.StateAParentinvolvementSurvey.com”) 


Make sure the link is clear and easily reproducible by parents. To do this, you will likely need to 
adjust the link that is automatically generated through your online survey program. Some survey 
programs allow you to make these adjustments from within the program, but with other programs, 
you may have to convert the link to a shorter link, for example, by using sites such as TinyUr| 


(https://tinyurl.com/) or Bitly (https://bitly.com 


Keep links as short as possible. 
Use a link that will make sense to parents rather than a random combination of letters and 
numbers. 
Avoid hidden and unintended characters in the links. 
=" Hidden characters include underscores (_ ), which may appear to be a space, rather than 
a character, if the link is underlined 
=" Unintended characters include periods when the web link is at the end of the sentence, 
because parents may mistake them for part of the web link. 


A review of several states’ paper surveys that included a printed web link revealed that the length of the 
links varied between 16 and 68 characters, and some included potentially hidden or unintended characters, 
as well as combinations of letters and numbers without meaning. Consider this fictional example of a 
difficult original online survey link, representative of what many states use: 


Original link: http://www.sunnyvale_SpecialEd.com/4762=Parent.htm 


Simplified link: www.SUSDParentSurvey.com 


The second simplified link is 26 characters shorter than the first; includes no hidden characters; and is clear, 
meaningful, and easily reproducible by parents. 
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Pilot the survey 


After you have developed your survey, you should pre-test, or pilot, it. Piloting the survey is a critical step to ensure 
development of a high-quality survey instrument. During the pilot, you should ask experts to review the survey, and you 
should administer your survey to a sample of potential respondent parents to gather information about the 
appropriateness of the survey’s elements. These elements include 


e Survey items, including content, wording, and response options 
e Survey organization, including length, format, and layout 


e Survey functionality (e.g., to test skip patterns for paper and online surveys, to ensure that any special design 
features work in online surveys) 


e Survey administration procedures 


There are several methods you can use to pilot a survey. You may want to use more than one, as each method will 
provide you with different feedback. 


e Expert review. There are two types of experts that you may want to consult: survey experts and content 
experts. A survey expert is someone who has expertise in survey design and administration. Survey experts can 
review your survey to ensure that it is technically sound. Content experts have expertise related to parent 
involvement or other topics included in your survey. Content experts can review your survey to ensure it covers 
the most important aspects of parent involvement and that the questions are worded in a way that will make 
sense to respondents. You might have both types of experts on your team, or you may want to consult a third- 
party contractor or stakeholders such as Parent Center staff. 


e Cognitive interviews. Cognitive interviews can be very useful to determine how respondents interpret questions 
and whether the appropriate response options are provided. During a cognitive interview, an interviewer sits 
with one or a small group of potential respondents, such as parents, and asks for their feedback as they 
complete the survey in-person with the interviewer. The interviewer either asks parents specific questions as 
they take the survey, or the interviewer asks parents to talk through their thought process as they answer the 
questions. 


e Field testing. Field testing your survey means administering it to a sample of potential respondents to make sure 
that 


— They are able to answer your survey questions 

— The available response options are appropriate 

— The length and format of your survey works well 

— For online surveys, the survey functions as you intended™ 


During your field test, you should administer your survey under the same conditions that you plan to administer 
your actual survey (like a “dress rehearsal”). Once you receive the surveys back, double-check to see if you need 
to make any tweaks. 


— Analyze the data to examine whether respondents skipped some items more frequently than others 
(and therefore, you may need to reword these items) 

— Calculate the overall response rate and identify where you might need to focus your follow-up efforts 
when you administer the actual survey 

— Determine how well your data tracking system is working 


54 Carlson and D’Agostino, 2015. 
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Field testing also can include follow-up interviews with some respondents to get their opinions about the survey 
and any items they might have found to be confusing or problematic. 


Because piloting your survey is so important, be sure to include adequate time in your overall timeline to ensure that 
you have time to pilot and revise as needed before you actually administer your survey. 
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Revise the survey based on the pilot 


After you pilot your survey and analyze the data, you likely will need to revise some of the survey items and perhaps 
reorganize the survey in some way. In addition, you can use what you learned to improve and refine your processes and 


procedures for survey administration and data tracking. For example, consider these questions: 
e Were you able to track all of the surveys that you sent out and received? 
e Were you able to match parents who responded to the survey with their child’s demographic information? 
e Were you able to appropriately follow-up with parents who did not respond? 
You will want to think about whether any steps were difficult and fine-tune your processes so things run more smoothly 


when you administer your survey for the actual data collection. 


For more information or support with your survey, contact your IDC State Liaison. 
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Resources 
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Important Information About Sampling 


Sampling is an area where you will need staff—or a third-party contractor—with specific training, expertise, or 
experience. If you are not currently working with an individual with expertise in sampling, we recommend that you 
contact your IDC State Liaison to connect you with resources and individuals with expertise related to sampling 
procedures. Additionally, Box 12 lists several resources you can use to learn more about sampling. 


Box 12. Resources on sampling 


Blair, E., and Blair, J. (2014). Applied Survey Sampling. Washington, DC: Sage Publications. 
Fowler, F.J., Jr. (2013). Survey Research Methods. Washington, DC: Sage Publications. 


Kalton, G. (1983). Quantitative Applications in the Social Sciences: Introduction to Survey Sampling. Washington, DC: 
Sage Publications. 


It is beyond the scope of this toolkit to go into detail about sampling, so here we will touch on two important steps you 
should take when using sampling as part of your parent involvement data collection: 


e Develop a sampling plan 
e Periodically adjust the sampling plan 
Develop a sampling plan 


The first step in sampling is to develop a sampling plan that details the steps that you will take. In fact, states that are 
planning to use sampling for Indicator B8 must submit their sampling plan to the Office of Special Education Programs 
(OSEP) for approval. Your sampling plan should specify your intended sample size and the type of sampling procedures 
you plan to use. 


Selected types of sampling procedures. There are many different types of sampling procedures. Random sampling 
refers to samples that are drawn completely at random. Random sampling is often preferred because, in theory, random 
samples should be representative. Random sampling can be difficult to do, though, and sometimes random sampling 
might not result in a representative sample (a phenomenon known as “unhappy randomization”). To reduce the risk of 
this, you can use probability sampling procedures. Probability sampling involves random selection at some point. A 
probability sample is a sample in which every unit in the population (e.g., parent) has a known, greater-than-zero chance 
of being selected in the sample and this probability can be accurately determined. The combination of these traits 
makes it possible to produce unbiased estimates of population totals by weighting sampled units according to their 
probability of selection. One sampling technique that is not appropriate in the context of Indicator B8 data collection is 
purposeful sampling. Purposeful sampling is when you do not randomly select participants at any point and therefore 
you cannot assess the extent to which the sample is representative. Below, we briefly describe some random sampling 
and probability sampling procedures that might be appropriate for your Indicator B8 data collection. Again, we suggest 
working with someone with specific training in sampling. 


e Simple random sampling is a sampling procedure where all parents in the target population have the same 
probability of being selected to participate in the survey, regardless of the other parents that are selected. You 
can accomplish simple random sampling using computer programs or drawing out of a hat. 


e Systematic random sampling is generally easy to do and, under certain circumstances, can be more precise than 
simple random sampling. First, develop a numbered list of parents in random order. Then, depending on the 
sample size you have chosen, determine an interval size (e.g., every 25th parent) and select parents from the list 
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at this interval until the sample is the desired size. Use a random number table (or random number generator) 
to determine which parent to start with on the list. For this to work correctly, you must list parents in random 
order with respect to the specific characteristics you are measuring. 


e Stratified random sampling can be used to ensure that specific groups of parents are adequately represented in 
your sample. This procedure involves dividing the entire population of parents into groups based on selected 
variables (e.g., their child’s disability category or race/ethnicity), and then selecting a simple random sample or 
systematic random sample from within each group. 


e Cluster sampling is similar to stratified sampling, but instead of sampling subgroups within a target population, 
you are dividing the population into clusters (e.g., schools or districts), from which a sample of clusters is 
randomly selected. In one-stage cluster sampling, all parents within the sampled clusters would be selected. In 
two-stage cluster sampling, simple random sampling is used to select parents within each cluster. The numbers 
of parents selected from different clusters are not necessarily equal. 


With these sampling procedures in mind, determine whether you will establish your sample by sampling parents of 
students with disabilities, by sampling districts within the state, or a combination of both. 


Sample size. An important aspect of a sampling plan is to determine the sample size necessary for Indicator B8. Should 
you survey 10 percent of your target parents? 20 percent? The answer is complicated and depends on many factors, 
including the goals of the data collection, the size of the target population, how precise you want your results to be in 
comparison to the true population value, the response rate that you expect to obtain, and the sampling strategy you are 
using. If you are using simple random sampling, you can use one of the many sample size calculators that are available 
online. However, regardless of the sampling procedure you are using, we recommend that you obtain assistance from 
individuals with expertise related to sampling procedures. 


Box 13 presents important information about OSEP’s requirements for sampling for Indicator B8. 
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Box 13. OSEP requirements related to sampling for Indicator B8 


OSEP Requirements for Sampling for Indicator B8 
OSEP has specific requirements that states must follow when conducting sampling to ensure that the sample is representative. 
Specifically, 
Samples within a district must be representative of that specific district, with respect to the unique distribution of 
variables, such as disability category, age, race, and gender, within each district 
If your state chooses to sample districts, 
You must include districts with average daily memberships of more than 50,000 in the sample every year 


You can use sampling within some districts and use a census (i.e., obtaining data from the total targeted 
population of parents of students with disabilities) in other districts 


You must include each district at least once over the course of the State Performance Plan (SPP) period 
You must collect data from a representative sample of districts each year 


OSEP Requirements for Reporting on Sampling for Indicator B8 
In addition to fulfilling the requirements related to how to sample, states must report to OSEP some important information 
related to their sampling procedures: 
The sampling procedures you followed (e.g., random/stratified, forms validation) 
Similarities or differences of the sample to the population of families with children with disabilities (e.g., how the 
sample will represent all aspects of the population, such as disability category, race, age, gender, etc.) at the state level 
Similarities or differences of the sample to the population of families with children with disabilities (e.g., how the 
sample will represent all aspects of the population, such as disability category, race, age, gender, etc.) at the district 
level 
— NOTE: When collecting data from small districts, do not report any information on performance that would result 
in the disclosure of personally identifiable information about individual children or where the available numbers 
are too small to yield statistically reliable information. 


Source: U.S. Department of Education 2017. 2018 Part B SPP/APR Instructions. 


Periodically adjust the sampling plan 


Once you develop a sampling plan, you will need to adjust it periodically, especially if the demographics across your 
state shifted since you determined your sampling plan. Changing demographics across or within certain districts could 
mean you need to examine your sampling strategy to ensure the representativeness of the sample. For example, 


e Across and within districts, have there been changes, additions, deletions, or redefinitions? 
e Arethere extremely large districts or small districts that did not exist before? 
e Have the sizes of districts changed? 


You may also need to adjust your sampling plan if your necessary sample size changes or if your response rates have 
changed over the years. 


What States Are Doing 


e Several states use sampling plans where all districts participate in the Indicator B8 data collection process every other year 
instead of every 5 years. This approach allows each district within the state to receive results on its parent involvement ina 
timely manner, which enables districts and states to better determine whether and how improvement activities they 
implement are affecting parent involvement. 

e Several states use stratified random sampling, stratifying districts by student enrollment, region, race/ethnicity, and 
socioeconomic level to ensure the results are representative of the state as a whole. 

e One state uses cluster sampling with a two-stage process, whereby the state samples districts and then randomly selects 
parents within those sampled districts. This state also adjusted its sampling plan to reflect the current demographics in the 
most recent Child Count data. 
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Additional Information About the NCSEAM Scale 


The NCSEAM Scale 


The NCSEAM scale measures schools’ efforts to facilitate parent involvement and is recognized as a high-quality tool for 
collecting data related to Indicator B8.°° The items and the cut-point scores used for the calculation of state 
performance on Indicator B8 were developed with input from families of children with disabilities. The scale meets 
rigorous measurement and scoring standards, and the developers piloted the items with a nationally representative 
sample of parents of students with disabilities in six states.°° High scores on the scale indicate that parents perceive 
schools as effectively promoting parent involvement. 


Rasch Analysis and the NCSEAM Scale 


The NCSEAM scale was developed through the Rasch measurement framework, which is an approach to measure 
development that is designed to create a valid measure of a given construct with unidimensional scales whose metrics 
are true interval data. In the case of Indicator B8, the construct is schools’ facilitation of parent involvement, as 
perceived by parents. With interval data, scoring units are consistent, such that the difference between a score of 500 
and 600, for example, is the same as the difference between a score of 600 and 700.°’ 


Rasch analysis yields a value for each item that estimates its location on the measurement scale, corresponding to its 
empirically estimated agreeability. An item’s location is also referred to as its calibration value. Items located lower on 
the scale are items that parents consistently agree with more, while items located higher on the scale are those that 
parents tend to agree with less. The relative location of items on the scale does not change depending on the particular 
parents who respond to the survey. Moreover, a parent who agrees with an item has a high likelihood of agreeing even 
more strongly with the items below it on the scale. The Data Accountability Center provides the following example to 
illustrate how this works: A person taking a math test will find that some items are easier than others. For example, 
regardless of how well this person does on the math test, he or she is more likely to get the easier items correct (for 
example, addition) than the difficult ones (for example, calculus). This would likely hold true for anyone taking the test. 
Elbaum (2012, p. 3) explains the model as follows: 


Data are analyzed in such a way that there is a well-defined relationship between item locations (referred to as item 
calibrations) and person locations (referred to as person measures). Items located higher on the scale require more of 
the attribute being measured to be successfully completed as indicated by a correct response to an item or a high 
agreement with a survey item.”* 


The NCSEAM Item Bank 


Most states use a 25-item version of the scale that NCSEAM recommended in 2006, although there are 78 items that 
were calibrated and states can select items from this “item bank.”°? The use of the items in the item bank is based on 
the Rasch framework. As mentioned above, Rasch analysis yields a value for each item that estimates its location on the 
measurement scale (calibration value). From a measurement perspective, items with similar locations (calibration 
values) are basically interchangeable.© Therefore, you can start with the 25-item version of the survey and substitute an 
item from that version with an item from the item bank with a similar location. However, the developers recommend 


55 Elbaum 2014. 

56 Elbaum et al. 2011. 

57 Data Accountability Center 2008. 
58 Elbaum 2012, p. 3. 

59 Elbaum 2014. 

60 Data Accountability Center 2008. 
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that you use at least 25 items and only substitute items with similar locations. Ideally, the selected items will reflect the 
whole range of the scale, from the most agreeable to the least agreeable items. 


The NCSEAM Cut-Point Score 


To determine what score on the NCSEAM scale would reflect parents who felt that their child’s school was adequately 
facilitating parent involvement, NCSEAM convened a group of nationally representative stakeholders, including parents 
of children with disabilities, state directors of special education, state early intervention coordinators, district and 
program personnel, advocates, attorneys, and community representatives.® This group addressed the question of how 
high a score had to be to “count” as a positive result for Indicator B8. The stakeholder team recommended using a 
measure of 600 as the standard for a positive response. This score corresponds to the item calibration for the item, “The 
school explains what options parents have if they disagree with a decision of the school.” A score of 600 would mean 
that the parent had a .95 likelihood of responding “agree,” “strongly agree,” or “very strongly agree” to that item. 


What States Are Doing 


Hawaii explained in their 2014 APR that they selected items from the NCSEAM item bank. They determined that a reliability of 
.90 or above could be achieved with the 25 items that they selected. The item with the highest location (or calibration value) 
Hawaii chose was, “In preparation for my child’s transition planning meeting | was given information about the options my child 
will have after high school.” If a parent agreed with this item, there was a high likelihood that the same parent would agree with 
the other 24 items with lower calibrations. The item with the lowest location (or calibration value) that Hawaii chose was, “At the 
IEP meeting, we discussed accommodations and modifications my child would need.” Items at the bottom of the scale such as 
this one are items that parents more likely tend to agree with. Hawaii used Rasch analysis to score the NCSEAM, which resulted 
in a score for each parent who responded to the survey. Each parent’s score is then compared to the cut-point score of 600. 


61 |bid. 
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